CIHealth Run Log

Environment: DEV

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:57:43	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 599dd2c · merge 599dd2c	n/a	n/a	passed	#4433 (merged)	0	n/a
23:32:28	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 421438d · merge 9baa329	n/a	provision	failed	⚠#4461 (merged)	1	provision (1) resource not ready, name: acrpull, kind: Deployment, status: Unknown context deadline exceeded 23:32:28 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/acrpull · suite step graph Show raw failure time=2026-03-17T00:03:10.626Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull description="Step acrpull\n Kind: Helm\n" time=2026-03-17T00:03:11.206Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull values="map[image:mcr.microsoft.com/aks/msi-acrpull@sha256:042511defb0842f02593e48e68d83a20a1af4265db4e23fca484356e34a6c883 namespace:acrpull replicas:2 ttlRotationFraction:0.5]" time=2026-03-17T00:03:11.206Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-03-17T00:03:11.515Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull namespace=acrpull time=2026-03-17T00:03:11.808Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull dryRun=false time=2026-03-17T00:03:11.808Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-03-17T00:03:11.923Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-03-17T00:08:19.495Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull err="resource not ready, name: acrpull, kind: Deployment, status: Unknown\ncontext deadline exceeded" time=2026-03-17T00:08:19.495Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-03-17T00:08:19.495Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-03-17T00:08:23.270Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull release=acrpull namespace=acrpull status=failed description="Release \"acrpull\" failed: resource not ready, name: acrpull, kind: Deployment, status: Unknown\ncontext deadline exceeded" values="map[image:mcr.microsoft.com/aks/msi-acrpull@sha256:042511defb0842f02593e48e68d83a20a1af4265db4e23fca484356e34a6c883 namespace:acrpull replicas:2 ttlRotationFraction:0.5]" time=2026-03-17T00:08:23.272Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yUzW7TQBCA736KkU+JZEdJK/ET1ENJOaGiqiAORRzWuxMzzXrG7I5bRcrDo8QNXVoS0oijZ7/95metyTwqBIzSBYsRzsAZNWoqjwOpbtHqR2I3jRqI6wL60CfT4O8Q3iFvIrE1dhsefssAAPJZF1Wa6wf9Bc6JSUk4LyA3NrSd9xWxI67j6OF71JANEmWuIyvNGsyLl8uaSOU/hdfiMblcWmEN4j2G0qNxGEr0aJ+kSC+/79Md5iirZ/TWdYGtl2WDrH8cPuOuxF0Kk0rYz301npxR4vrcNRQjCV+JJ7tMCk4KVVkglwF/dhi1bDdkWu2R2og2oJZNp2bT/38UhzuyWBprpeODMsx8FxXD7hffAe9/4Ze0csDMj/H8fcjHmfZONVV+7tHzntz/L86E51RfmnZztq2glxLGdJx2g6aO7Pu7bNFVGBgV44f1qonZCu5/YEBQajCqaVqoUO8RGQbOKK7Dg/xkfPKqHJ+Wk9dfxuPp+HQ6mdzkQxiNYCfzZjp5e5MPh9kKboUYFsTujJgxJAtSGB73YroQn27CbAVtkPXpY6EFtOJ6+DBJAV7qAhqM0dRrpQSHAapl0rzDaH8FAAD//wniDsXGBQAA" time=2026-03-17T00:08:23.274Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-03-17T00:08:23.274Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull err="error running Helm release deployment Step, failed to deploy helm release: resource not ready, name: acrpull, kind: Deployment, status: Unknown\ncontext deadline exceeded"
23:25:16	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 2c15e11 · merge 2c15e11	n/a	unknown	failed	#4486 (merged)	1	non-artifact-backed failure (no junit artifacts)
23:01:46	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 0b90840 · merge 04f43f0	n/a	e2e	failed	#4482 (merged)	7	e2e (7) timeout during CreateHCPClusterAndWait; context deadline exceeded 23:01:46 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapErrors \| 0xc00141f530>: failed to create HCP cluster cluster-zstream-4-20-g88rq7, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2, error: failed waiting for cluster="cluster-zstream-4-20-g88rq7" in resourcegroup="rg-zstream-upgrade-4-20-qg77d2" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2, error: context deadline exceeded { msg: "failed to create HCP cluster cluster-zstream-4-20-g88rq7, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2, error: failed waiting for cluster=\"cluster-zstream-4-20-g88rq7\" in resourcegroup=\"rg-zstream-upgrade-4-20-qg77d2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013b6420>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2", }, <fmt.wrapErrors \| 0xc00141f500>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-g88rq7\" in resourcegroup=\"rg-zstream-upgrade-4-20-qg77d2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013b6420>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapErrors \| 0xc00141f530>: failed to create HCP cluster cluster-zstream-4-20-g88rq7, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2, error: failed waiting for cluster="cluster-zstream-4-20-g88rq7" in resourcegroup="rg-zstream-upgrade-4-20-qg77d2" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2, error: context deadline exceeded { msg: "failed to create HCP cluster cluster-zstream-4-20-g88rq7, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2, error: failed waiting for cluster=\"cluster-zstream-4-20-g88rq7\" in resourcegroup=\"rg-zstream-upgrade-4-20-qg77d2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013b6420>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2", }, <fmt.wrapErrors \| 0xc00141f500>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-g88rq7\" in resourcegroup=\"rg-zstream-upgrade-4-20-qg77d2\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013b6420>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-g88rq7 in resource group rg-zstream-upgrade-4-20-qg77d2", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 23:01:46 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc0010ec0e0>: failed to create HCP cluster cluster-zstream-4-19-jlc5tn: failed starting cluster creation "cluster-zstream-4-19-jlc5tn" in resourcegroup="rg-zstream-upgrade-4-19-rdkr4c": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rdkr4c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jlc5tn -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-jlc5tn: failed starting cluster creation \"cluster-zstream-4-19-jlc5tn\" in resourcegroup=\"rg-zstream-upgrade-4-19-rdkr4c\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rdkr4c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jlc5tn\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010ec0c0>{ msg: "failed starting cluster creation \"cluster-zstream-4-19-jlc5tn\" in resourcegroup=\"rg-zstream-upgrade-4-19-rdkr4c\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rdkr4c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jlc5tn\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001310300>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "956b09c4-e7a8-41ec-a551-201abc827d34", ], "Date": [ "Mon, 16 Mar 2026 23:51:33 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["9547"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc0010ec000>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j8172544.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rdkr4c/pr... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc0010ec0e0>: failed to create HCP cluster cluster-zstream-4-19-jlc5tn: failed starting cluster creation "cluster-zstream-4-19-jlc5tn" in resourcegroup="rg-zstream-upgrade-4-19-rdkr4c": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rdkr4c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jlc5tn -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-jlc5tn: failed starting cluster creation \"cluster-zstream-4-19-jlc5tn\" in resourcegroup=\"rg-zstream-upgrade-4-19-rdkr4c\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rdkr4c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jlc5tn\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010ec0c0>{ msg: "failed starting cluster creation \"cluster-zstream-4-19-jlc5tn\" in resourcegroup=\"rg-zstream-upgrade-4-19-rdkr4c\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rdkr4c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-jlc5tn\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001310300>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "956b09c4-e7a8-41ec-a551-201abc827d34", ], "Date": [ "Mon, 16 Mar 2026 23:51:33 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["9547"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc0010ec000>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j8172544.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-rdkr4c/pr... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 23:01:46 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:400]: Unexpected error: <exported.ResponseError \| 0xc0014fa090>: GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-tm62sq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["3208"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "9ea98782-3e1d-4675-95fc-91b3b30cbc83", ], "Date": [ "Mon, 16 Mar 2026 23:51:50 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc00116ad40>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j8172544.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-tm62sq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Accept": ["application/json"], "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "Authorization": ["redacted"], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-16T23:51:27Z\"}", ], }, Body: nil, GetBody: nil, ContentLength: 0, TransferEncoding: nil, Close: false, Host: "rp.j8172544.hcpsvc.osadev.cloud", Form: nil, PostForm: nil, MultipartForm: nil, Trailer: nil, RemoteAddr: "", RequestURI: "", TLS: nil, Cancel: nil, Response: nil, Pattern: "", ctx: <suppressed context>, pat: nil, matches: nil, otherValues: nil, }, TLS: { Version: 772, HandshakeComplete: true, DidResume: false, CipherSuite: 4865, CurveID: 29, NegotiatedProtocol: "h2", NegotiatedProtocolIsMutual: true, ServerName: "rp.j8172544.hcpsvc.osadev.cl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:400]: Unexpected error: <exported.ResponseError \| 0xc0014fa090>: GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-tm62sq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["3208"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "9ea98782-3e1d-4675-95fc-91b3b30cbc83", ], "Date": [ "Mon, 16 Mar 2026 23:51:50 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc00116ad40>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j8172544.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-tm62sq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Accept": ["application/json"], "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "Authorization": ["redacted"], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-16T23:51:27Z\"}", ], }, Body: nil, GetBody: nil, ContentLength: 0, TransferEncoding: nil, Close: false, Host: "rp.j8172544.hcpsvc.osadev.cloud", Form: nil, PostForm: nil, MultipartForm: nil, Trailer: nil, RemoteAddr: "", RequestURI: "", TLS: nil, Cancel: nil, Response: nil, Pattern: "", ctx: <suppressed context>, pat: nil, matches: nil, otherValues: nil, }, TLS: { Version: 772, HandshakeComplete: true, DidResume: false, CipherSuite: 4865, CurveID: 29, NegotiatedProtocol: "h2", NegotiatedProtocolIsMutual: true, ServerName: "rp.j8172544.hcpsvc.osadev.cl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 23:01:46 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc0006678a0>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed starting cluster creation "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-rrmlfs": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-rrmlfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed starting cluster creation \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-rrmlfs\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-rrmlfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0006677e0>{ msg: "failed starting cluster creation \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-rrmlfs\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-rrmlfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000aa3800>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["3945"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "415a824d-9904-4ac6-84ed-d1fb2146174a", ], "Date": [ "Mon, 16 Mar 2026 23:51:54 GMT", ], "Content-Length": ["107"], }, Body: <exported.nopClosingBytesReader \| 0xc000667640>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j8172544.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-rrmlfs/providers/Mi... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc0006678a0>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed starting cluster creation "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-rrmlfs": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-rrmlfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed starting cluster creation \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-rrmlfs\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-rrmlfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0006677e0>{ msg: "failed starting cluster creation \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-rrmlfs\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-rrmlfs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000aa3800>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["3945"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "415a824d-9904-4ac6-84ed-d1fb2146174a", ], "Date": [ "Mon, 16 Mar 2026 23:51:54 GMT", ], "Content-Length": ["107"], }, Body: <exported.nopClosingBytesReader \| 0xc000667640>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j8172544.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-rrmlfs/providers/Mi... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 23:01:46 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error: <errors.joinError \| 0xc00179b4a0>: failed to cleanup resource group: failed listing hcp clusters in resourcegroup="external-auth-cluster-wdzps7": GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-wdzps7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <fmt.wrapError \| 0xc0012c71a0>{ msg: "failed to cleanup resource group: failed listing hcp clusters in resourcegroup=\"external-auth-cluster-wdzps7\": GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-wdzps7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0012c7180>{ msg: "failed listing hcp clusters in resourcegroup=\"external-auth-cluster-wdzps7\": GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-wdzps7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0010f29c0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "78f9e056-a4a6-4a4c-9502-b92baf96f96a", ], "Date": [ "Mon, 16 Mar 2026 23:50:41 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["2880"], }, Body: <exported.nopClosingBytesReader \| 0xc0012c70c0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error: <errors.joinError \| 0xc00179b4a0>: failed to cleanup resource group: failed listing hcp clusters in resourcegroup="external-auth-cluster-wdzps7": GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-wdzps7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <fmt.wrapError \| 0xc0012c71a0>{ msg: "failed to cleanup resource group: failed listing hcp clusters in resourcegroup=\"external-auth-cluster-wdzps7\": GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-wdzps7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0012c7180>{ msg: "failed listing hcp clusters in resourcegroup=\"external-auth-cluster-wdzps7\": GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-wdzps7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0010f29c0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "78f9e056-a4a6-4a4c-9502-b92baf96f96a", ], "Date": [ "Mon, 16 Mar 2026 23:50:41 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["2880"], }, Body: <exported.nopClosingBytesReader \| 0xc0012c70c0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 23:01:46 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:247]: Unexpected error: <fmt.wrapError \| 0xc0011ba3a0>: failed waiting for hcpCluster="admin-cred-lifecycle-7vk4vx" in resourcegroup="admin-credential-lifecycle-test-w9rq7p" to finish getting creds: GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/1f808343-553b-459a-bd27-b8d61314ce16 -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-7vk4vx\" in resourcegroup=\"admin-credential-lifecycle-test-w9rq7p\" to finish getting creds: GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/1f808343-553b-459a-bd27-b8d61314ce16\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0007ef7d0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["2901"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "311609f5-9d4d-4266-8e59-b7148893c123", ], "Date": [ "Mon, 16 Mar 2026 23:50:27 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc0011ba2c0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j8172544.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/1f808343-553b-459a-bd27-b8d61314ce16", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-16T23:50:07Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Authorization": [... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:247]: Unexpected error: <fmt.wrapError \| 0xc0011ba3a0>: failed waiting for hcpCluster="admin-cred-lifecycle-7vk4vx" in resourcegroup="admin-credential-lifecycle-test-w9rq7p" to finish getting creds: GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/1f808343-553b-459a-bd27-b8d61314ce16 -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-7vk4vx\" in resourcegroup=\"admin-credential-lifecycle-test-w9rq7p\" to finish getting creds: GET https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/1f808343-553b-459a-bd27-b8d61314ce16\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0007ef7d0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["2901"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "311609f5-9d4d-4266-8e59-b7148893c123", ], "Date": [ "Mon, 16 Mar 2026 23:50:27 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc0011ba2c0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j8172544.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/1f808343-553b-459a-bd27-b8d61314ce16", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-16T23:50:07Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Authorization": [... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 23:01:46 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000730520>: failed to create HCP cluster cluster-zstream-4-21-x4bct2: failed starting cluster creation "cluster-zstream-4-21-x4bct2" in resourcegroup="rg-zstream-upgrade-4-21-zbvp7q": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-zbvp7q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-x4bct2 -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-x4bct2: failed starting cluster creation \"cluster-zstream-4-21-x4bct2\" in resourcegroup=\"rg-zstream-upgrade-4-21-zbvp7q\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-zbvp7q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-x4bct2\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000730500>{ msg: "failed starting cluster creation \"cluster-zstream-4-21-x4bct2\" in resourcegroup=\"rg-zstream-upgrade-4-21-zbvp7q\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-zbvp7q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-x4bct2\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000988180>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["3181"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "5408bd3e-ffe3-4a8e-a5cb-61938c9b2d00", ], "Date": [ "Mon, 16 Mar 2026 23:51:51 GMT", ], "Content-Length": ["107"], }, Body: <exported.nopClosingBytesReader \| 0xc000730160>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j8172544.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-zbvp7q/pr... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000730520>: failed to create HCP cluster cluster-zstream-4-21-x4bct2: failed starting cluster creation "cluster-zstream-4-21-x4bct2" in resourcegroup="rg-zstream-upgrade-4-21-zbvp7q": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-zbvp7q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-x4bct2 -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-x4bct2: failed starting cluster creation \"cluster-zstream-4-21-x4bct2\" in resourcegroup=\"rg-zstream-upgrade-4-21-zbvp7q\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-zbvp7q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-x4bct2\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000730500>{ msg: "failed starting cluster creation \"cluster-zstream-4-21-x4bct2\" in resourcegroup=\"rg-zstream-upgrade-4-21-zbvp7q\": PUT https://rp.j8172544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-zbvp7q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-x4bct2\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000988180>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["3181"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "5408bd3e-ffe3-4a8e-a5cb-61938c9b2d00", ], "Date": [ "Mon, 16 Mar 2026 23:51:51 GMT", ], "Content-Length": ["107"], }, Body: <exported.nopClosingBytesReader \| 0xc000730160>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "PUT", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j8172544.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-zbvp7q/pr... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
22:20:32	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 0b90840 · merge 04f43f0	n/a	provision	failed	★#4482 (merged)	1	provision (1) "arohcp-monitor" has no deployed releases 22:20:32 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite step graph Show raw failure time=2026-03-16T22:48:23.283Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-16T22:48:23.954Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:4d4a8749-1fac-4d4f-916f-0e98fd14766f]] prometheusSpec:map[externalLabels:map[cluster:prow-j3650688-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j3650688-svc-ezhn.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-557258f7768e454a9792a5c6bd539df9/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T22:48:23.955Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-16T22:48:24.319Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus time=2026-03-16T22:48:24.730Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false time=2026-03-16T22:48:24.730Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-16T22:48:26.065Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus numVersions=0 time=2026-03-16T22:48:26.177Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="\"arohcp-monitor\" has no deployed releases" time=2026-03-16T22:48:26.177Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-16T22:48:26.177Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-16T22:48:26.288Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed to get status for release arohcp-monitor: release: not found" time=2026-03-16T22:48:26.288Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-16T22:48:26.289Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: \"arohcp-monitor\" has no deployed releases"
22:02:22	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 38f866a · merge 38f866a	n/a	provision	failed	#4454 (merged)	1	provision (1) failed post-install: Unauthorized 22:02:22 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph Show raw failure time=2026-03-16T22:45:32.131Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-03-16T22:45:32.681Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-16T22:45:32.681Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:45:32.965Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-03-16T22:45:33.075Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-03-16T22:45:33.075Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:45:33.182Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:47:08.180Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: Unauthorized" time=2026-03-16T22:47:08.180Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:47:08.180Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:47:18.981Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed to get status for release mce-config: kubernetes cluster unreachable: the server has asked for the client to provide credentials" time=2026-03-16T22:47:18.981Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:47:18.981Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
21:58:41	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8cb4bd5 · merge 8cb4bd5	n/a	n/a	passed	#4466 (merged)	0	n/a
21:57:18	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 454b8c0 · merge 2c15e11	n/a	provision	failed	#4486 (merged)	1	provision (1) tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\ 21:57:18 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph Show raw failure time=2026-03-16T22:43:56.374Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-03-16T22:43:56.974Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-16T22:43:56.974Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:43:57.031Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-03-16T22:43:57.145Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-03-16T22:43:57.145Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:43:57.251Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:45:27.656Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69b883de32e3a9000188c8bc.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)" time=2026-03-16T22:45:27.656Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:45:27.656Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:45:33.046Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config release=mce-config namespace=multicluster-engine status=pending-install description="Initial install underway" values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-16T22:45:33.048Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6SUMW/bMBCFd/2KgyYbkIJUaVJARYY07RS0Q9OpRQeKvNCMqTuCpBwY8I8vaFmR6lRJkK7P732+x7MvsxjBY+DOSwxwCUpEEUVjccHNPcp4Y0jVIXpDuoBe+iZafJRwg7RXghNykJe/MgCA/Bb9xki8kpI7inkB+dp2IaK3GEuhFFMpmaJna9GXmyo52s5GI3tbiaQNYV7M4bSXpWNr5PaAC+IFxnWvfmeLyckOqRycrSChsUWKtfay7sGHMXuLP8QG2s1jnatku2a6MzoZLEthh9Q/hOP4mNSWG2GnX/J1P5Y6TH6L++arrUMfVuYuTsYe+GEanzT+ZEgZ0s8Vf3lBb+OmB508YjMmBt4RKK32yXqPos/seYb2esBndJa3afb//t3+jXpSakS9leM8O6FF5FlC9vtjtu4a9IQRw5f0pw3ZDh5W6BGiaTFE0TpoMD4gEiyUiJjkRV6dVhfl6Vn57uJHVdXvz+rzDz/zJZycwKznvK6SZ5nt4J4NwdqQujRE6CenhgnGCzM9Lcc3JduB85w+HQctwLHqza+DFGBZF9BiCEInJHuFHprtpLzCIP8EAAD//+XobDEQBQAA" time=2026-03-16T22:45:33.050Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T22:45:33.050Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce-config\\\": Post \\\"https://guard.69b883de32e3a9000188c8bc.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (delete jobs.batch finalize-mce-config)"
21:27:34	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head ad2d12b · merge 8550ece	n/a	provision	failed	#4219 (merged)	1	provision (1) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError("HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 21:27:34 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-03-16T21:53:36.176Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-03-16T21:53:59.173Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nFailed to log in to ACR arohcpsvcdev: ERROR: The command failed with an unexpected error. Here is the traceback:\nERROR: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 204, in _new_conn\n sock = connection.create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/connection.py\", line 60, in create_connection\n for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/python3.12/socket.py\", line 963, in getaddrinfo\n for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 787, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 488, in _make_request\n raise new_e\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 464, in _make_request\n self._validate_conn(conn)\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 1093, in _validate_conn\n conn.connect()\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 759, in connect\n self.sock = sock = self._new_conn()\n ^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NameResolutionError(self.host, self, e) from e\nurllib3.exceptions.NameResolutionError: HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 841, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/retry.py\", line 535, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/knack/cli.py\", line 233, in invoke\n cmd_result = self.invocation.execute(args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 682, in execute\n raise ex\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 812, in _run_jobs_serially\n results.append(self._run_job(expanded_arg, cmd_copy))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 781, in _run_job\n result = cmd_copy(params)\n ^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 336, in __call__\n return self.handler(args, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py\", line 120, in handler\n return op(command_args)\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/custom.py\", line 294, in acr_login\n login_server, _, password = get_login_credentials(\n ^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 505, in get_login_credentials\n return _get_credentials(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 449, in _get_credentials\n return login_server, EMPTY_GUID, _get_aad_token(cli_ctx,\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 285, in _get_aad_token\n return _get_aad_token_after_challenge(cli_ctx,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 159, in _get_aad_token_after_challenge\n response = requests.post(url=authhost, data=urlencode(content), headers=headers,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 115, in post\n return request(\"post\", url, data=data, json=json, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n return session.request(method=method, url=url, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, *kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTo check existing issues, please visit: https://github.com/Azure/azure-cli/issues\n exit status 1"
21:19:12	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 730aedc	n/a	provision	failed	#4157 (open)	1	provision (1) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError("HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 21:19:12 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-velero-image · suite step graph Show raw failure time=2026-03-16T22:01:59.625Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image description="Step mirror-velero-image\n Kind: ImageMirror\n From {<nil> velero.server.registry <nil>}:{<nil> velero.server.repository <nil>}@{<nil> velero.server.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-03-16T22:02:24.194Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-velero-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nFailed to log in to ACR arohcpsvcdev: ERROR: The command failed with an unexpected error. Here is the traceback:\nERROR: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 204, in _new_conn\n sock = connection.create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/connection.py\", line 60, in create_connection\n for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/python3.12/socket.py\", line 963, in getaddrinfo\n for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 787, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 488, in _make_request\n raise new_e\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 464, in _make_request\n self._validate_conn(conn)\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 1093, in _validate_conn\n conn.connect()\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 759, in connect\n self.sock = sock = self._new_conn()\n ^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NameResolutionError(self.host, self, e) from e\nurllib3.exceptions.NameResolutionError: HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 841, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/retry.py\", line 535, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/knack/cli.py\", line 233, in invoke\n cmd_result = self.invocation.execute(args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 682, in execute\n raise ex\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 812, in _run_jobs_serially\n results.append(self._run_job(expanded_arg, cmd_copy))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 781, in _run_job\n result = cmd_copy(params)\n ^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 336, in __call__\n return self.handler(args, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py\", line 120, in handler\n return op(command_args)\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/custom.py\", line 294, in acr_login\n login_server, _, password = get_login_credentials(\n ^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 505, in get_login_credentials\n return _get_credentials(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 449, in _get_credentials\n return login_server, EMPTY_GUID, _get_aad_token(cli_ctx,\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 276, in _get_aad_token\n token_params = _handle_challenge_phase(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 100, in _handle_challenge_phase\n challenge = requests.get(request_url, verify=not should_disable_connection_verify())\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 73, in get\n return request(\"get\", url, params=params, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n return session.request(method=method, url=url, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, *kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTo check existing issues, please visit: https://github.com/Azure/azure-cli/issues\n exit status 1"
21:16:20	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 599dd2c · merge 599dd2c	n/a	n/a	passed	#4433 (merged)	0	n/a
21:14:45	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 38f866a · merge 38f866a	n/a	provision	failed	★#4454 (merged)	1	provision (1) "arohcp-monitor" has no deployed releases 21:14:45 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/prometheus · suite step graph Show raw failure time=2026-03-16T21:47:01.740Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-16T21:47:02.459Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:c0e2dd57-8426-4558-9dfd-5350ad0d2166]] prometheusSpec:map[externalLabels:map[cluster:prow-j1712640-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j1712640-mgmt-1-frzu.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-3693fddb11a14071bdf1db60a24df604/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j1712640-mgmt-1-frzu.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-511f0b56c3044481b3f308eb487d8d5d/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T21:47:02.459Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:02.799Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-16T21:47:03.148Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-16T21:47:03.148Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:03.768Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus numVersions=0 time=2026-03-16T21:47:03.926Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="\"arohcp-monitor\" has no deployed releases" time=2026-03-16T21:47:03.926Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:03.926Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:04.083Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed to get status for release arohcp-monitor: release: not found" time=2026-03-16T21:47:04.083Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:04.083Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: \"arohcp-monitor\" has no deployed releases"
21:13:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head dcec1a0	n/a	provision	failed	#4393 (open)	1	provision (1) failed post-install: Unauthorized 21:13:55 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/prometheus · suite step graph Show raw failure time=2026-03-16T21:45:19.279Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-16T21:45:19.891Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:00d715af-ce8d-440f-8c1b-630fda34474f]] prometheusSpec:map[externalLabels:map[cluster:prow-j9755136-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j9755136-mgmt-1-77iw.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-8392ee99bb7b4c47be651b9c93097b71/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j9755136-mgmt-1-77iw.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-ce8065f976124c569ca5653ccdbdf2a0/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T21:45:19.891Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:45:20.256Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-16T21:45:20.682Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-16T21:45:20.682Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:45:20.833Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:28.992Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed post-install: Unauthorized" time=2026-03-16T21:47:28.992Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:28.992Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:45.274Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed post-install: Unauthorized" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:00d715af-ce8d-440f-8c1b-630fda34474f]] prometheusSpec:map[externalLabels:map[cluster:prow-j9755136-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j9755136-mgmt-1-77iw.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-8392ee99bb7b4c47be651b9c93097b71/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j9755136-mgmt-1-77iw.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-ce8065f976124c569ca5653ccdbdf2a0/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T21:47:45.275Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yWTW/aTBDH7/4UK5+C5EUJz9M0osqhjXqqUlVp1UqpeljvTs0G7441MwYh8eErYwIuhAYZjjv7n9/OwLw4KUEUAWNNFljdKmfEiMlLuMD8Cax88tGNWcjHIlOt6bMJsDHBDOLKwpWxz+bBz0QppdK7smYBesAS0kylhnBiKx0wekHS0zoHzWIEdAAhb7kRpdmLvhVhAJlAzRorICNIB9QffHQ+Fmdw6mq/oLtv497XdE9r9cH7f6sf6r1sTQkkwURTAA2pLqEPZHrDre/QYhTjI5AOEJAW2ho7gfMiic8cI89NdV7iHGnqY6EZROcL6fejNvXbORcQgUzZ/0/a4UV0oAksUlOaJ2EpggDrTaOfhmFBMkWvoulAFiwQtKk8A81gr6360HowXuvNr0Azb6HT+0dNsWNAB6bTQdf31mId5fQYzkM4Ovz7Woz4WPyAfII4vcP42xc1GfEYd4edC555bX/2/25K704itA737RB5NXNtaxYMm2ZZ3dkV4uUE/8YHswExSBM2r71Dq2hfbAs2+fUu2Zbxx2aZcrJU8wkQKPGheT1UKgeZA0R14YxAY75IR5eja335n766/ja6Gv//Zjy6fEwHajhUBzVvx6Obx3QwSJbqCX1UUx/drY8RqPMJgFFtN3935e/u+mSpKsLmdhtopip0rfg4SKZKLDIVgNkUDRLJAal80UneAds/AQAA//+NZhXbqAgAAA==" time=2026-03-16T21:47:45.275Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:47:45.276Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
21:02:46	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c5356b7	n/a	provision	failed	★#4483 (open)	1	provision (1) failed to prepare kubeconfig: failed to ensure cluster admin role: failed to get current user object ID: failed to get current user: /me request is only valid with delegated authentication flow. 21:02:46 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RouteMonitorOperator/management/default-ingresscontroller-hack · suite step graph Show raw failure time=2026-03-16T21:37:32.736Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=default-ingresscontroller-hack description="Step default-ingresscontroller-hack\n Kind: Shell\n Command: make default-ingresscontroller-hack\n" time=2026-03-16T21:37:34.160Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=default-ingresscontroller-hack err="failed to prepare kubeconfig: failed to ensure cluster admin role: failed to get current user object ID: failed to get current user: /me request is only valid with delegated authentication flow."
20:39:21	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 7e50507 · merge 56bf314	n/a	provision	failed	#4485 (merged)	1	provision (1) tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\ 20:39:21 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-policies · suite step graph Show raw failure time=2026-03-16T21:18:18.396Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies description="Step deploy-policies\n Kind: Helm\n" time=2026-03-16T21:18:19.181Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies values=map[] time=2026-03-16T21:18:19.181Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-16T21:18:19.367Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies namespace=open-cluster-management-policies time=2026-03-16T21:18:19.647Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies dryRun=false time=2026-03-16T21:18:19.647Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-16T21:18:19.757Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-16T21:18:22.140Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies err="unable to continue with install: could not get information about the resource Policy \"ingress-controller-check\" in namespace \"open-cluster-management-policies\": an error on the server (\"Internal Server Error: \\\"/apis/policy.open-cluster-management.io/v1/namespaces/open-cluster-management-policies/policies/ingress-controller-check\\\": Post \\\"https://guard.69b86fad54541c00013118c8.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (get policies.policy.open-cluster-management.io ingress-controller-check)" time=2026-03-16T21:18:22.140Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-16T21:18:22.140Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-16T21:18:22.250Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies err="failed to get status for release policy: release: not found" time=2026-03-16T21:18:22.250Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies time=2026-03-16T21:18:22.250Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-policies err="error running Helm release deployment Step, failed to deploy helm release: unable to continue with install: could not get information about the resource Policy \"ingress-controller-check\" in namespace \"open-cluster-management-policies\": an error on the server (\"Internal Server Error: \\\"/apis/policy.open-cluster-management.io/v1/namespaces/open-cluster-management-policies/policies/ingress-controller-check\\\": Post \\\"https://guard.69b86fad54541c00013118c8.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \\\"crypto/rsa: verification error\\\" while trying to verify candidate authority certificate \\\"ca\\\")\") has prevented the request from succeeding (get policies.policy.open-cluster-management.io ingress-controller-check)"
20:38:15	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0f991f0	n/a	unknown	failed	#4484 (closed)	1	non-artifact-backed failure (no junit artifacts)
20:28:44	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head f1bd325	n/a	provision	failed	⚠#4467 (closed)	1	provision (1) failed post-install: an error on the server ("Internal Server Error: \\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce\\": Post \\"<url>": proxy error from localhost:9443 while dialing guard.69b86d9039445e00019c3745.svc.cluster.local:443 20:28:44 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce · suite step graph Show raw failure time=2026-03-16T21:05:37.584Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce description="Step deploy-mce\n Kind: Helm\n" time=2026-03-16T21:05:38.143Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f imageRegistry:arohcpsvcdev.azurecr.io imageRootRepository:acm-d-cache]" time=2026-03-16T21:05:38.144Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T21:05:38.255Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce namespace=multicluster-engine time=2026-03-16T21:05:38.365Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce dryRun=false time=2026-03-16T21:05:38.365Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T21:05:38.473Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T21:08:00.581Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce\\\": Post \\\"https://guard.69b86d9039445e00019c3745.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69b86d9039445e00019c3745.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (delete jobs.batch finalize-mce)" time=2026-03-16T21:08:00.581Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T21:08:00.581Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T21:08:01.250Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="failed to get status for release mce: kubernetes cluster unreachable: the server has asked for the client to provide credentials" time=2026-03-16T21:08:01.250Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T21:08:01.250Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce\\\": Post \\\"https://guard.69b86d9039445e00019c3745.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69b86d9039445e00019c3745.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (delete jobs.batch finalize-mce)"
20:25:42	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 138964d · merge 138964d	n/a	provision	failed	#4226 (merged)	1	provision (1) failed post-install: Unauthorized 20:25:42 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/prometheus · suite step graph Show raw failure time=2026-03-16T21:01:24.308Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-16T21:01:24.970Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:0c3e5218-5ebd-4cb6-aca0-9a4ce3c51e56]] prometheusSpec:map[externalLabels:map[cluster:prow-j6922880-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j6922880-mgmt-1-hrjs.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-a6c994f4b3b644e9a187298c90941783/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j6922880-mgmt-1-hrjs.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-1cd0089dd64c44c8a8571eaf7bd7f61d/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T21:01:24.970Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:01:25.281Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-16T21:01:25.615Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-16T21:01:25.615Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:01:25.781Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:02:18.525Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed post-install: Unauthorized" time=2026-03-16T21:02:18.525Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:02:18.525Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:02:32.544Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed post-install: Unauthorized" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:0c3e5218-5ebd-4cb6-aca0-9a4ce3c51e56]] prometheusSpec:map[externalLabels:map[cluster:prow-j6922880-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j6922880-mgmt-1-hrjs.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-a6c994f4b3b644e9a187298c90941783/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j6922880-mgmt-1-hrjs.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-1cd0089dd64c44c8a8571eaf7bd7f61d/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T21:02:32.545Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6xWTW8TMRC976+wcmqkOGqCqKqgHqBwQkWoIJCKOHjtx8bN2mONZxsi5cejZPuxDYSmSY/79s2bZ489nqKGKEamhi2yOlPOiBFT1jii8hpWPvroJlnYx2qgWuiTCbiHcIO4RnIy9g7u/yiUUqr3mdwFRS/EvYHqJaYAmaLJm1+37HvksqnxmKNNDZZgoqnAQ25q7CMyO81t7NBSFOMjWNvU6CabCjrDUnRZC4mpX0g9IBAvtDV2ipeV5PxSO3ArmOcmHaSYyGmaR/xV7F1UmhK6810hgk29f6WbEhwhyNqkdKDC/f04TCYLsan2OgYdkUUWBG2Sz+CbvfY6koPG70QsB9ylRyr7xkfInHi2Z/j/vH8B33iLTvsxTFObdGiR9sRlMQIdIOztzkIdF5TA5h/trQ09r5ss4Etq3e9kYEvslpxPsreQ3vnofKye7eqbqb0z4mP1HeWUaHZO8ZevGjbiKW62bBd8zrf4xna+tZaaKIfXpSO0e13Wpi/admeCuUukM2S1tqztmhFaRutofe82kj8r63ukmhYBB636scjTuYufb4qH7vFh9VjnYqnmUzCU+IAsJiRVQuZAVEfOCFbwUW98PD7Rx6/06OTreDQ5Hk3Gr696fTUcqq2c8WR0etXr94uluiYf1cxHd+ZjBHdGDIrqYbLojhSbs0SxVIlp9ffB6EAlci15N5GBqqkaqIC8euKLpSJ2YFUuOot3yPZPAAAA//9TChsdCAkAAA==" time=2026-03-16T21:02:32.545Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T21:02:32.546Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
20:21:40	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 89dbea9 · merge 0d2c310	n/a	provision	failed	#4224 (merged)	1	provision (1) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError("HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 20:21:40 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RouteMonitorOperator/global/mirror-blackbox-image · suite step graph Show raw failure time=2026-03-16T21:01:18.829Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=global step=mirror-blackbox-image description="Step mirror-blackbox-image\n Kind: ImageMirror\n From {<nil> routeMonitorOperator.blackboxExporterImage.registry <nil>}:{<nil> routeMonitorOperator.blackboxExporterImage.repository <nil>}@{<nil> routeMonitorOperator.blackboxExporterImage.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-03-16T21:01:41.643Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=global step=mirror-blackbox-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nFailed to log in to ACR arohcpsvcdev: ERROR: The command failed with an unexpected error. Here is the traceback:\nERROR: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 204, in _new_conn\n sock = connection.create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/connection.py\", line 60, in create_connection\n for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/python3.12/socket.py\", line 963, in getaddrinfo\n for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 787, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 488, in _make_request\n raise new_e\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 464, in _make_request\n self._validate_conn(conn)\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 1093, in _validate_conn\n conn.connect()\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 759, in connect\n self.sock = sock = self._new_conn()\n ^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NameResolutionError(self.host, self, e) from e\nurllib3.exceptions.NameResolutionError: HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 841, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/retry.py\", line 535, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/knack/cli.py\", line 233, in invoke\n cmd_result = self.invocation.execute(args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 682, in execute\n raise ex\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 812, in _run_jobs_serially\n results.append(self._run_job(expanded_arg, cmd_copy))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 781, in _run_job\n result = cmd_copy(params)\n ^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 336, in __call__\n return self.handler(args, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py\", line 120, in handler\n return op(command_args)\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/custom.py\", line 294, in acr_login\n login_server, _, password = get_login_credentials(\n ^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 505, in get_login_credentials\n return _get_credentials(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 449, in _get_credentials\n return login_server, EMPTY_GUID, _get_aad_token(cli_ctx,\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 285, in _get_aad_token\n return _get_aad_token_after_challenge(cli_ctx,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 159, in _get_aad_token_after_challenge\n response = requests.post(url=authhost, data=urlencode(content), headers=headers,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 115, in post\n return request(\"post\", url, data=data, json=json, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n return session.request(method=method, url=url, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, *kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTo check existing issues, please visit: https://github.com/Azure/azure-cli/issues\n exit status 1"
20:20:33	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head ad2d12b · merge 8550ece	n/a	provision	failed	#4219 (merged)	1	provision (1) Unauthorized 20:20:33 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/acrpull · suite step graph Show raw failure time=2026-03-16T21:01:50.368Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull description="Step acrpull\n Kind: Helm\n" time=2026-03-16T21:01:51.040Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull values="map[image:mcr.microsoft.com/aks/msi-acrpull@sha256:042511defb0842f02593e48e68d83a20a1af4265db4e23fca484356e34a6c883 namespace:acrpull replicas:2 ttlRotationFraction:0.5]" time=2026-03-16T21:01:51.040Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-03-16T21:01:51.336Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull namespace=acrpull time=2026-03-16T21:01:51.639Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull dryRun=false time=2026-03-16T21:01:51.639Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-03-16T21:01:51.761Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-03-16T21:01:59.535Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull err=Unauthorized time=2026-03-16T21:01:59.536Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-03-16T21:01:59.536Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-03-16T21:02:02.487Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull release=acrpull namespace=acrpull status=failed description="Release \"acrpull\" failed: Unauthorized" values="map[image:mcr.microsoft.com/aks/msi-acrpull@sha256:042511defb0842f02593e48e68d83a20a1af4265db4e23fca484356e34a6c883 namespace:acrpull replicas:2 ttlRotationFraction:0.5]" time=2026-03-16T21:02:02.488Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5yTT2vbQBDF7/oUg042WMZ2aaAuOaR/TqUlpKWHlB5WuxN14t0ZdXeUEPCHL7KceiscN+Rmz775zRsxr/CoEDFJFy0mOAdn1KipPU6kvkWrn4jdOmkkbmYwlL6YgH9LeIe8q6TW2Mfy9EcBAFB+xXhHFi+slY61nEFpbGw77//5OWjfd0klXO2tfMAbYlISzqQ1sSNu0nz/fx7IRklyo3MroRe+BBYSVf8H+i4pxivx+G5ozFCVFdYo3mOs6sPrY2/f9ITao3EYK/RoR/by5tMTR4zcwYj13XhyRombCxcoJRK+FE/2IRuQgRPaiFqFTs0O3O60xzbMvs5xk7n4UtxnYVKJpw/iSbcjmyob5Cri7w6T7k3m857LOb7uy0i7s6/McPdHkcXPt8WmqzEyKqaPfYpSsYX7XxgRlAImNaGFGvUekWHijGJfnpSrxeqsWryqlmffVsv1Yrl+vbwupzCfwynNm+tyOi22cCvEsCF258SMMcu+MBwin2d9HPJiC22U/vVgdAatuEH8PMgMvDQzCJiSaXqkRIcR6odseYfJ/gkAAP//JisWsaEEAAA=" time=2026-03-16T21:02:02.489Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-03-16T21:02:02.489Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull err="error running Helm release deployment Step, failed to deploy helm release: Unauthorized"
19:52:40	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8cb4bd5 · merge 8cb4bd5	n/a	n/a	passed	#4466 (merged)	0	n/a
19:51:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 10f6db8 · merge 532823b	n/a	provision	failed	⚠#4468 (merged)	1	provision (1) failed to apply namespace openshift-route-monitor-operator: Unauthorized 19:51:36 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RouteMonitorOperator/management/deploy · suite step graph Show raw failure time=2026-03-16T20:16:22.096Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-16T20:16:22.596Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy values="map[imageDigestBlackbox:sha256:b04a9fef4fa086a02fc7fcd8dcdbc4b7b35cc30cdee860fdc6a19dd8b208d63e imageDigestManager:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageDigestOperator:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageRegistryBlackbox:arohcpsvcdev.azurecr.io imageRegistryManager:arohcpsvcdev.azurecr.io imageRegistryOperator:arohcpsvcdev.azurecr.io imageRepositoryBlackbox:prometheus/blackbox-exporter imageRepositoryManager:app-sre/route-monitor-operator imageRepositoryOperator:app-sre/route-monitor-operator]" time=2026-03-16T20:16:22.596Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T20:16:23.346Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace openshift-route-monitor-operator: Unauthorized"
19:44:06	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 0b90840 · merge 04f43f0	n/a	provision	failed	⚠#4482 (merged)	1	provision (1) failed post-install: warning: Hook post-install multicluster-engine/templates/finalize-mce.job.yaml failed: an error on the server ("Internal Server Error 19:44:06 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce · suite step graph Show raw failure time=2026-03-16T20:26:47.029Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce description="Step deploy-mce\n Kind: Helm\n" time=2026-03-16T20:26:47.648Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f imageRegistry:arohcpsvcdev.azurecr.io imageRootRepository:acm-d-cache]" time=2026-03-16T20:26:47.648Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T20:26:47.707Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce namespace=multicluster-engine time=2026-03-16T20:26:47.819Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce dryRun=false time=2026-03-16T20:26:47.819Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T20:26:47.932Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T20:28:26.817Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="failed post-install: warning: Hook post-install multicluster-engine/templates/finalize-mce.job.yaml failed: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce?fieldManager=templatize&fieldValidation=Strict&force=false\\\": Post \\\"https://guard.69b863f2318b900001096102.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69b863f2318b900001096102.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (patch jobs.batch finalize-mce)" time=2026-03-16T20:28:26.817Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T20:28:26.817Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T20:28:26.961Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="failed to get status for release mce: query: failed to query with labels: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/multicluster-engine/secrets?labelSelector=name%3Dmce%2Cowner%3Dhelm\\\": Post \\\"https://guard.69b863f2318b900001096102.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69b863f2318b900001096102.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (get secrets)" time=2026-03-16T20:28:26.961Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T20:28:26.961Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: warning: Hook post-install multicluster-engine/templates/finalize-mce.job.yaml failed: an error on the server (\"Internal Server Error: \\\"/apis/batch/v1/namespaces/multicluster-engine/jobs/finalize-mce?fieldManager=templatize&fieldValidation=Strict&force=false\\\": Post \\\"https://guard.69b863f2318b900001096102.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69b863f2318b900001096102.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (patch jobs.batch finalize-mce)"
19:39:49	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 205984e · merge 205984e	n/a	provision	failed	★#4449 (merged)	1	provision (1) unable to continue with install: could not get information about the resource ClusterRole "route-monitor-opera-98lwvxopk1c4mcyxq48sgl92b67gbgycw0kkd4rdsr5" in namespace "": Unauthorized 19:39:49 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RouteMonitorOperator/management/deploy · suite step graph Show raw failure time=2026-03-16T20:05:53.635Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-16T20:05:54.216Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy values="map[imageDigestBlackbox:sha256:b04a9fef4fa086a02fc7fcd8dcdbc4b7b35cc30cdee860fdc6a19dd8b208d63e imageDigestManager:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageDigestOperator:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageRegistryBlackbox:arohcpsvcdev.azurecr.io imageRegistryManager:arohcpsvcdev.azurecr.io imageRegistryOperator:arohcpsvcdev.azurecr.io imageRepositoryBlackbox:prometheus/blackbox-exporter imageRepositoryManager:app-sre/route-monitor-operator imageRepositoryOperator:app-sre/route-monitor-operator]" time=2026-03-16T20:05:54.216Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T20:05:54.534Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy namespace=openshift-route-monitor-operator time=2026-03-16T20:05:54.815Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy dryRun=false time=2026-03-16T20:05:54.815Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T20:05:54.921Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T20:05:58.181Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="unable to continue with install: could not get information about the resource ClusterRole \"route-monitor-opera-98lwvxopk1c4mcyxq48sgl92b67gbgycw0kkd4rdsr5\" in namespace \"\": Unauthorized" time=2026-03-16T20:05:58.181Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T20:05:58.181Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T20:05:58.291Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="failed to get status for release route-monitor-operator: release: not found" time=2026-03-16T20:05:58.291Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T20:05:58.291Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: unable to continue with install: could not get information about the resource ClusterRole \"route-monitor-opera-98lwvxopk1c4mcyxq48sgl92b67gbgycw0kkd4rdsr5\" in namespace \"\": Unauthorized"
19:20:04	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head ad2d12b · merge 8550ece	n/a	provision	failed	#4219 (merged)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 19:20:04 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-16T19:59:43.936Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-16T20:00:56.397Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=a296ceb2-a43e-4725-9169-304a0c3e17b6 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=2e022157-7935-49d1-9846-1981212496c7 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj0348032.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj0348032.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j0348032 -n arohcp-prow-dbcs-j0348032 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0348032/providers/Microsoft.Network/dnszones/j0348032.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j0348032-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j0348032-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j0348032-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"95e11a6d-913e-4adb-9890-9e0ebd50bfa2\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j0348032-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0348032-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j0348032-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
19:18:02	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4834b3b	n/a	provision	failed	#4467 (closed)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 19:18:02 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-16T19:59:44.540Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-16T20:01:14.547Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=d47d6e1b-67d5-43b3-8321-cc3d6e757e42 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=a4766fe2-bcce-4945-9d82-6beb905a1e30 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj3980288.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj3980288.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j3980288 -n arohcp-prow-dbcs-j3980288 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3980288/providers/Microsoft.Network/dnszones/j3980288.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j3980288-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j3980288-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j3980288-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"7fac78f7-d715-4063-b78f-ca3c8ffeb1ef\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j3980288-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3980288-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j3980288-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
19:14:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 205984e · merge 205984e	n/a	provision	failed	#4449 (merged)	1	provision (1) failed post-install: Unauthorized 19:14:08 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/prometheus · suite step graph Show raw failure time=2026-03-16T19:29:50.886Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-16T19:29:51.578Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:a76595ab-de64-4e87-a475-d43fd793c26e]] prometheusSpec:map[externalLabels:map[cluster:prow-j0558976-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j0558976-mgmt-1-xnqn.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-855a3699732143c49a482e7b776f3104/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j0558976-mgmt-1-xnqn.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-9011cef126304e52a16fcc6c1686f3bb/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T19:29:51.578Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T19:29:51.916Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-16T19:29:52.287Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-16T19:29:52.287Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T19:29:52.463Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T19:30:50.509Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed post-install: Unauthorized" time=2026-03-16T19:30:50.509Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T19:30:50.509Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T19:30:57.439Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed post-install: Unauthorized" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:a76595ab-de64-4e87-a475-d43fd793c26e]] prometheusSpec:map[externalLabels:map[cluster:prow-j0558976-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j0558976-mgmt-1-xnqn.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-855a3699732143c49a482e7b776f3104/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j0558976-mgmt-1-xnqn.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-9011cef126304e52a16fcc6c1686f3bb/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T19:30:57.441Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yWz27TQBDG736KlU+NlI3SVK1oUA9QOKGiqiCQijisdwdnG+/OanbcEKkPjxw3jVvqYOwcPZ75ebx/5vuSAlgQRCxJQxQXwihWrLICjjC7A82frDfzyGR9PhZ16LNy8BSCe/CbSAxKb8OjH4kQQqRfgO6thndaY+k5HYtUES50kA69ZSS5LDOQkRWDdMBkdaySAqEDXkAZ03EbaJcjMQApRvr/0tcrLosyMtANFvDeemN93rnz/YiWnrdFHyAUuHYwaKWeQzqv0jdVWKPY+vw7ZAvE5SX6XzYvSbFF/wKljLMxPsa3hGs0V3WvXdb4uvX9/uybsoCX3RRA7JRXOdCEygL6QPTmdyVBgcoA9UHk4IFU0buF5ZtY1040elbWA0kHDmkttdILOCyS4oF7jCsVDktcIS2tz2UEltmaD7ao23E3CBfQSFx5aLlP+ynVXW48ezQgCTRSNSn6H6COI+KfEPLAEKUKYSDhSViGYSIjqbzXfjUg68jgpAo2At333rdntB6MzV7D74DEA8bVM0rP+t4f7ywqHSktKryZyFf1UFFObc90NQ4qndrObFdn1If/cVf+EuEhAt5BuVuzm0mPRmS4C+pjfxrSPOzjr2p8Ww/Jz7fJ7tJ8rKxiTB7EagEEgq2DyMoFkQGvALw4MoqhCh+ls+nsTE5P5PHZ1+Pz+ex8fjq7TUdiMhFtOSfT+en0Nh2Nkgdxh9aLpfXmwnoP1DC46MXO1zYN7UsnmzyIQFi93TU6FgFNndwNMhYF5mPhIEaVV0gkAySydePnDUT9JwAA///iuuQVhgsAAA==" time=2026-03-16T19:30:57.442Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T19:30:57.443Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
19:09:30	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3207545	n/a	unknown	failed	#4045 (open)	1	non-artifact-backed failure (no junit artifacts)
19:09:09	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8f711fa	n/a	unknown	failed	#4095 (open)	1	non-artifact-backed failure (no junit artifacts)
19:09:03	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head dcec1a0	n/a	unknown	failed	#4393 (open)	1	non-artifact-backed failure (no junit artifacts)
19:08:56	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c6d34c6	n/a	provision	failed	#4457 (open)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 19:08:56 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-16T19:45:32.200Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-16T19:45:58.890Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=6ad615cf-74ad-45b9-98fd-af2a5733844d && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=f52e7edf-bae6-411f-bce8-6a372bd4b676 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj4297088.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj4297088.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j4297088 -n arohcp-prow-dbcs-j4297088 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4297088/providers/Microsoft.Network/dnszones/j4297088.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j4297088-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j4297088-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j4297088-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"b6de3e02-8f2f-4bf8-9489-ac97663eecaa\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j4297088-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4297088-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j4297088-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
19:00:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8cb4bd5 · merge 8cb4bd5	n/a	unknown	failed	#4466 (merged)	1	non-artifact-backed failure (no junit artifacts)
18:58:57	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 38f866a · merge 38f866a	n/a	n/a	passed	#4454 (merged)	0	n/a
18:58:42	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 7b9532a · merge 7b9532a	n/a	provision	failed	★#4455 (merged)	1	provision (1) unable to continue with install: could not get information about the resource ServiceAccount "metrics-proxy-sa" in namespace "<namespace>": Unauthorized 18:58:42 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Agent/management/deploy · suite step graph Show raw failure time=2026-03-16T19:20:03.896Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-16T19:20:04.528Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy values="map[azure:map[clientId:4044d85c-56be-4b90-b184-215a1e0a5de8 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-prow-maestro-j6821504.westus3-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j6821504-mgmt-1 credsKeyVault:map[name:ah-prow-mg-j6821504-1 secret:hcp-underlay-j6821504-mgmt-1] deployment:map[zoneCount:4] glog_v:2 image:map[digest:sha256:391b002a10ffd994fa4f47e4d1096bac91e512769c0073df782f3cd8d8fe43e7 registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:3 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:331aaa0d553e8eec9b69261f1dc6fd74eadc1daec5907287dc5e9bbc3a7e2bc0 registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]" time=2026-03-16T19:20:04.528Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-16T19:20:05.320Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy namespace=maestro time=2026-03-16T19:20:05.606Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy dryRun=false time=2026-03-16T19:20:05.606Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-16T19:20:06.157Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-16T19:20:08.352Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy err="unable to continue with install: could not get information about the resource ServiceAccount \"metrics-proxy-sa\" in namespace \"maestro\": Unauthorized" time=2026-03-16T19:20:08.352Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-16T19:20:08.352Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-16T19:20:08.461Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy err="failed to get status for release maestro-agent: release: not found" time=2026-03-16T19:20:08.461Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-03-16T19:20:08.462Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: unable to continue with install: could not get information about the resource ServiceAccount \"metrics-proxy-sa\" in namespace \"maestro\": Unauthorized"
18:39:43	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c5356b7	n/a	unknown	failed	#4483 (open)	1	non-artifact-backed failure (no junit artifacts)
18:39:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 10f6db8 · merge 532823b	n/a	provision	failed	⚠#4468 (merged)	1	provision (1) failed to apply namespace aro-hcp: Unauthorized 18:39:14 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.Backend/service/deploy · suite step graph Show raw failure time=2026-03-16T19:20:18.975Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-16T19:20:19.506Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpprowoidcj1092352.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpprowoidcj1092352.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:2076d051-d5ae-4edc-b8d1-25ff9f8a0c39 databaseName:arohcpprow-rp-j1092352 databaseUrl:https://arohcpprow-rp-j1092352.documents.azure.com:443/ location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-r7hdm913/pipeline@sha256:9e3e8e42d97e5eec67d63504a6b1d5967d7ea2b8942b4a88ad9fc6730bc3d19c replicas:2 zoneCount:4] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1] maestro:map[sourceEnvironmentIdentifier:arohcpprow] miMock:map[certName:msiMockCert2 clientId:e8723db7-9b9e-46a4-9f7d-64d75c3534f0 principalId:d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-r7hdm913/pipeline:pull workloadIdentityClientId:8ddc6831-4500-4dc8-93f7-0660f64237ec workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:2076d051-d5ae-4edc-b8d1-25ff9f8a0c39 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]" time=2026-03-16T19:20:19.506Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy time=2026-03-16T19:20:20.138Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace aro-hcp: Unauthorized"
18:38:32	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head b359186	n/a	provision	failed	⚠#4305 (closed)	1	provision (1) unable to continue with install: could not get information about the resource Deployment "secrets-store-sync-controller-manager" in namespace "<namespace>": Unauthorized 18:38:32 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/management/deploy · suite step graph Show raw failure time=2026-03-16T19:14:49.008Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-16T19:14:49.695Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:490c3b795cb12bd9e5a3680bb7dabd7ea02799c30b14ed960611517c5d808e6b imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[cpu:50m memory:100Mi] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-03-16T19:14:49.695Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-16T19:14:49.894Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy namespace=secret-sync-controller time=2026-03-16T19:14:50.213Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy dryRun=false time=2026-03-16T19:14:50.213Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-16T19:14:50.322Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-16T19:14:53.586Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy err="unable to continue with install: could not get information about the resource Deployment \"secrets-store-sync-controller-manager\" in namespace \"secret-sync-controller\": Unauthorized" time=2026-03-16T19:14:53.586Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-16T19:14:53.586Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-16T19:14:53.694Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy err="failed to get status for release secret-sync-controller: release: not found" time=2026-03-16T19:14:53.694Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy time=2026-03-16T19:14:53.694Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: unable to continue with install: could not get information about the resource Deployment \"secrets-store-sync-controller-manager\" in namespace \"secret-sync-controller\": Unauthorized"
18:22:33	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 3f58874 · merge 138964d	n/a	provision	failed	#4226 (merged)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 18:22:33 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-16T19:00:09.988Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-16T19:00:24.322Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=36a2b479-629f-4bdd-822e-834cb3abc063 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=fd72cb74-fdc0-40a7-b8ae-1d206ba66b31 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj7738240.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj7738240.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j7738240 -n arohcp-prow-dbcs-j7738240 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7738240/providers/Microsoft.Network/dnszones/j7738240.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j7738240-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j7738240-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j7738240-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"a9667be0-9e5b-481c-b8bf-3b7eb1890100\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j7738240-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7738240-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j7738240-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nError from server (InternalError): error when retrieving current configuration of:\nResource: \"/v1, Resource=namespaces\", GroupVersionKind: \"/v1, Kind=Namespace\"\nName: \"clusters-service\", Namespace: \"\"\nfrom server for: \"STDIN\": an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/clusters-service\\\": Post \\\"https://guard.69b84f6a54541c000131187e.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69b84f6a54541c000131187e.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (get namespaces clusters-service)\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
18:11:05	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 89dbea9 · merge 0d2c310	n/a	n/a	passed	#4224 (merged)	0	n/a
17:47:31	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 7b9532a · merge 7b9532a	n/a	provision	failed	#4455 (merged)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 17:47:31 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-16T18:28:51.532Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-16T18:29:53.534Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=ebf9c2e8-f4e4-455e-93fc-72cb59ea132b && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=adb25dcd-b00c-4348-8ad9-25c8e21b1f51 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj6026880.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj6026880.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j6026880 -n arohcp-prow-dbcs-j6026880 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6026880/providers/Microsoft.Network/dnszones/j6026880.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j6026880-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j6026880-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j6026880-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"b602f500-ed9f-4351-a0d5-0538cad403c3\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j6026880-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6026880-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j6026880-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
17:47:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 38f866a · merge 38f866a	n/a	provision	failed	★#4454 (merged)	1	provision (1) failed to run ARM step: failed to poll previously-existing deployment: timed out waiting for pre-existing deployment to finish 17:47:08 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/management/output · suite step graph Show raw failure time=2026-03-16T18:28:02.813Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=management step=output description="Step output\n Kind: ARM\n Template: ../dev-infrastructure/templates/output-mgmt.bicep\n Parameters: ../dev-infrastructure/configurations/output-mgmt.tmpl.bicepparam" time=2026-03-16T18:28:03.295Z level=DEBUG+2 msg="Waiting for existing deployment to complete" serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=management step=output deployment=4d5cdb6c661fd7a2d0d2ebc735e3a3b5a279f779ac9ba11949d31a9b27d78ed9 time=2026-03-16T18:31:04.547Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=management step=output err="failed to run ARM step: failed to poll previously-existing deployment: timed out waiting for pre-existing deployment to finish"
17:29:49	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 6b3c5b8 · merge 8b7576c	n/a	unknown	failed	#4473 (merged)	1	non-artifact-backed failure (no junit artifacts)
17:23:01	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 3b95084 · merge 205984e	n/a	provision	failed	#4449 (merged)	1	provision (1) make[2]: *** [Makefile:32: deploy] Error 1 17:23:01 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-16T18:03:17.403Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-16T18:03:50.165Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=767aa9f5-c1a3-468f-a085-c445483009cf && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=8160dccb-38e2-4b9a-a40a-66e5cdb22ae3 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj2702464.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj2702464.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j2702464 -n arohcp-prow-dbcs-j2702464 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2702464/providers/Microsoft.Network/dnszones/j2702464.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j2702464-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j2702464-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j2702464-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"fa7c648c-f2f7-442d-b2a1-27c3deaa829a\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j2702464-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2702464-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j2702464-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: 3 errors occurred:\n\t* Unauthorized\n\t* Unauthorized\n\t* Unauthorized\n\n\nmake[2]: *** [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
17:06:53	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 599dd2c · merge 599dd2c	n/a	provision	failed	#4433 (merged)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 17:06:53 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-16T17:33:01.743Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-16T17:33:49.040Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=0028379a-fe01-4056-b086-40cc00b73848 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=7a657233-48ff-4136-83db-075edbbd38a0 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj4505088.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj4505088.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j4505088 -n arohcp-prow-dbcs-j4505088 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4505088/providers/Microsoft.Network/dnszones/j4505088.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j4505088-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j4505088-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j4505088-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"20783b0c-a764-4c66-a123-d16f025c284e\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j4505088-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4505088-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j4505088-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
16:53:38	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head ae94c93 · merge 8550ece	n/a	provision	failed	#4219 (merged)	1	provision (1) failed to apply namespace multicluster-engine: Unauthorized 16:53:38 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-crds · suite step graph Show raw failure time=2026-03-16T17:30:06.934Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds description="Step deploy-mce-crds\n Kind: Helm\n" time=2026-03-16T17:30:07.452Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds values=map[] time=2026-03-16T17:30:07.452Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-03-16T17:30:08.243Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace multicluster-engine: Unauthorized"
16:35:38	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 89dbea9 · merge 0d2c310	n/a	provision	failed	★#4224 (merged)	1	provision (1) failed to prepare kubeconfig: failed to ensure cluster admin role: failed to get current user object ID: failed to get current user: /me request is only valid with delegated authentication flow. 16:35:38 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RouteMonitorOperator/management/default-ingresscontroller-hack · suite step graph Show raw failure time=2026-03-16T17:00:54.253Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=default-ingresscontroller-hack description="Step default-ingresscontroller-hack\n Kind: Shell\n Command: make default-ingresscontroller-hack\n" time=2026-03-16T17:00:55.925Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=default-ingresscontroller-hack err="failed to prepare kubeconfig: failed to ensure cluster admin role: failed to get current user object ID: failed to get current user: /me request is only valid with delegated authentication flow."
16:30:17	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 3f58874 · merge 138964d	n/a	provision	failed	⚠#4226 (merged)	1	provision (1) "route-monitor-operator" has no deployed releases 16:30:17 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RouteMonitorOperator/management/deploy · suite step graph Show raw failure time=2026-03-16T16:55:02.720Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-16T16:55:03.362Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy values="map[imageDigestBlackbox:sha256:b04a9fef4fa086a02fc7fcd8dcdbc4b7b35cc30cdee860fdc6a19dd8b208d63e imageDigestManager:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageDigestOperator:sha256:5e94ff37bcb8c1ebebce0c7c1f33cef8e7cf163f1b0baf5b50c9b771ca112846 imageRegistryBlackbox:arohcpsvcdev.azurecr.io imageRegistryManager:arohcpsvcdev.azurecr.io imageRegistryOperator:arohcpsvcdev.azurecr.io imageRepositoryBlackbox:prometheus/blackbox-exporter imageRepositoryManager:app-sre/route-monitor-operator imageRepositoryOperator:app-sre/route-monitor-operator]" time=2026-03-16T16:55:03.362Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T16:55:03.643Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy namespace=openshift-route-monitor-operator time=2026-03-16T16:55:03.944Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy dryRun=false time=2026-03-16T16:55:03.944Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T16:55:04.673Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy numVersions=0 time=2026-03-16T16:55:04.782Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="\"route-monitor-operator\" has no deployed releases" time=2026-03-16T16:55:04.782Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T16:55:04.782Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T16:55:04.891Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="failed to get status for release route-monitor-operator: release: not found" time=2026-03-16T16:55:04.891Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy time=2026-03-16T16:55:04.891Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: \"route-monitor-operator\" has no deployed releases"
16:10:20	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 599dd2c · merge 599dd2c	n/a	provision	failed	★#4433 (merged)	1	provision (1) failed pre-install: warning: Hook pre-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/role.yaml failed: Unauthorized 16:10:20 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/prometheus · suite step graph Show raw failure time=2026-03-16T16:26:24.901Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-16T16:26:25.637Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:51a9932a-8ace-44d0-9e3f-f999b80392d3]] prometheusSpec:map[externalLabels:map[cluster:prow-j8137984-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j8137984-mgmt-1-63b2.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-9852b4015418490d932705247af74e0f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j8137984-mgmt-1-63b2.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-50d428240f7d4caaac3e60d6f99d85b5/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T16:26:25.638Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T16:26:26.084Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-03-16T16:26:26.477Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-03-16T16:26:26.477Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T16:26:26.616Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T16:26:47.452Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed pre-install: warning: Hook pre-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/role.yaml failed: Unauthorized" time=2026-03-16T16:26:47.452Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T16:26:47.452Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T16:27:00.535Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed pre-install: warning: Hook pre-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/role.yaml failed: Unauthorized" values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[customResourceState:map[config:map[spec:map[resources:[map[groupVersionKind:map[group:hypershift.openshift.io kind:HostedCluster version:v1beta1] labelsFromPath:map[_id:[spec clusterID] api_url:[spec services [service=APIServer] servicePublishingStrategy route hostname] name:[metadata name] namespace:[metadata namespace]] metricNamePrefix:hostedClusterAPI metrics:[map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=KubeAPIServerAvailable] status]] type:StateSet] help:KubeAPIServer availability condition name:kubeapiserver_available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Available] status]] type:StateSet] help:Cluster availability condition name:available] map[each:map[stateSet:map[labelName:status list:[True False Unknown] path:[status conditions [type=Progressing] status]] type:StateSet] help:Cluster progressing condition name:progressing]]]]]] create:true enabled:true key:config.yaml name:kube-state-metric-custom-resource-config] image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c] rbac:map[extraRules:[map[apiGroups:[hypershift.openshift.io] resources:[hostedclusters] verbs:[list watch]]]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:51a9932a-8ace-44d0-9e3f-f999b80392d3]] prometheusSpec:map[externalLabels:map[cluster:prow-j8137984-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j8137984-mgmt-1-63b2.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-9852b4015418490d932705247af74e0f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j8137984-mgmt-1-63b2.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-50d428240f7d4caaac3e60d6f99d85b5/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T16:27:00.536Z level=INFO msg="No resource info available in release, skipping resource diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus infoNil=false resourceCount=0 time=2026-03-16T16:27:00.536Z level=INFO msg="Kusto deep link for namespace-level troubleshooting" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus kustoLinkNamespace="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/3yOvaqDQBSEe59i2EpBRb0XL1zwDVKmsgnrOqgh+8PuBgn48MEUJlXgFIePGeZT1kS5GPqTnUKyYZ3pibhohii1w8C4kgbpKCN3nIqmatqi+inq9ly3/81+vchQlviS+f3rRZYdA0ZqBicVL/uHroNw3mrGmfcgkg3O2ytVfJvkuNkph7Pjq5JssH6kx/A4GGRQ+Ye7DOoZAAD///9xGUHfAAAA" time=2026-03-16T16:27:00.537Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-03-16T16:27:00.537Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed pre-install: warning: Hook pre-install hcp-prometheus/charts/kube-prometheus-stack/templates/prometheus-operator/admission-webhooks/job-patch/role.yaml failed: Unauthorized"
15:44:50	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 3cda2b6 · merge 8b7576c	n/a	provision	failed	⚠#4473 (merged)	1	provision (1) failed to apply namespace aro-hcp-admin-api: an error on the server ("Internal Server Error: \\"/api/v1/namespaces/aro-hcp-admin-api?fieldManager=templatize&force=false\\": Post \\"<url>": proxy error from localhost:9443 while dialing 15:44:50 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.AdminAPI/service/deploy · suite step graph Show raw failure time=2026-03-16T16:26:27.268Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-16T16:26:27.834Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy values="map[audit:map[connectSocket:false] azure:map[clientId:9b335788-9698-4c2e-b8b6-e774d43dd7a8 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] clustersService:map[namespace:clusters-service] cosmosDB:map[name:arohcpprow-rp-j8745984 url:https://arohcpprow-rp-j8745984.documents.azure.com:443/] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:admin-api-cert-prow-j8745984] deployment:map[limits:map[cpu:1 memory:1Gi] replicas:2 requests:map[cpu:200m memory:512Mi] zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 keyVaultName:aro-hcp-dev-svc-kv] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:admin-api-https] host:admin.j8745984.hcpsvc.osadev.cloud] image:map[digest:sha256:765bfaa0fea5572f7c547c3041dda88339289f0a850136bf92b4d880bb2d886f registry:arohcpsvcdev.azurecr.io repository:ci-op-yg9z9inx/pipeline] kusto:map[endpoint:https://hcp-dev-us-2.eastus2.kusto.windows.net] location:westus3 mise:map[deploy:false policyLabel:Geneva Actions] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-yg9z9inx/pipeline:pull workloadIdentityClientId:4e03dc43-0ff5-4d72-979f-67d7f7f56198 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:admin-api port:8443] serviceAccount:map[name:admin-api workloadIdentityClientId:163d854a-7dae-475f-9cc3-141cdae7b1c5 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] sessiongate:map[maxSessionTTL:24h minSessionTTL:1m mise:map[policyLabel:Session Gate] namespace:sessiongate servicePort:8080] virtualService:map[host:admin.j8745984.hcpsvc.osadev.cloud]]" time=2026-03-16T16:26:27.835Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy time=2026-03-16T16:26:28.066Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy namespace=aro-hcp-admin-api time=2026-03-16T16:26:28.357Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy dryRun=false time=2026-03-16T16:26:28.357Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy time=2026-03-16T16:26:28.537Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy time=2026-03-16T16:26:35.567Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy err="Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.37:15017, code 500: 500 Internal Server Error" time=2026-03-16T16:26:35.567Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy time=2026-03-16T16:26:35.567Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy time=2026-03-16T16:26:36.947Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy release=admin-api namespace=aro-hcp-admin-api status=failed description="Release \"admin-api\" failed: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.37:15017, code 500: 500 Internal Server Error" values="map[audit:map[connectSocket:false] azure:map[clientId:9b335788-9698-4c2e-b8b6-e774d43dd7a8 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] clustersService:map[namespace:clusters-service] cosmosDB:map[name:arohcpprow-rp-j8745984 url:https://arohcpprow-rp-j8745984.documents.azure.com:443/] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:admin-api-cert-prow-j8745984] deployment:map[limits:map[cpu:1 memory:1Gi] replicas:2 requests:map[cpu:200m memory:512Mi] zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 keyVaultName:aro-hcp-dev-svc-kv] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:admin-api-https] host:admin.j8745984.hcpsvc.osadev.cloud] image:map[digest:sha256:765bfaa0fea5572f7c547c3041dda88339289f0a850136bf92b4d880bb2d886f registry:arohcpsvcdev.azurecr.io repository:ci-op-yg9z9inx/pipeline] kusto:map[endpoint:https://hcp-dev-us-2.eastus2.kusto.windows.net] location:westus3 mise:map[deploy:false policyLabel:Geneva Actions] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-yg9z9inx/pipeline:pull workloadIdentityClientId:4e03dc43-0ff5-4d72-979f-67d7f7f56198 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:admin-api port:8443] serviceAccount:map[name:admin-api workloadIdentityClientId:163d854a-7dae-475f-9cc3-141cdae7b1c5 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] sessiongate:map[maxSessionTTL:24h minSessionTTL:1m mise:map[policyLabel:Session Gate] namespace:sessiongate servicePort:8080] virtualService:map[host:admin.j8745984.hcpsvc.osadev.cloud]]" time=2026-03-16T16:26:36.949Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5yTQWvbQBCF7/oVg04xaEPiUFNcckjTQqG0mNT0kNLDancsT7zaXWZGDgH/+CIbJ2qJjdPrm/e9mZF2ioAKjJI6dihwDd6qVVsHPEv1Azr9StFPRZliU8FO+m5bfJZwjXGrSLZuL49+FQAA5W3oRJHvUsCPFD3FpqygtL6laGwmIyhCKZq2abWvlNWOu3E860IYMLkLwdSDDE5m6bJ5ztqjX+bz2V3qFP/qdBT5gY5RZ5zW5JFvgxXp/YtsjUPW/2EHK7q8FVZiSJSSodgw9q59AK/J4bcUSRMf6HV0gC1/41zqor5l6S33qse0qExOTg44qeMnzCE9tfiWKQ8w2/9CC3JW0TAuGGWJfPQ7/yTWzobXpjbrA5sWvz8Uq65Gjqgon/uXLsUGHpfICEotito2Q436iBjhzFvFXj4rxxfjibm4MpeT+eVkOp5Mx+/vyxGcn8MRz9W7+3I0KjbwkCjCiqK/phiRB/eZIryc5fAe/z3EYgOZU199GbSCnPzOfFpIBSE1FbQoYps+MrFHhvppsLxHcX8CAAD//wRnC6JFBAAA" time=2026-03-16T16:26:36.952Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy time=2026-03-16T16:26:36.953Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy step=deploy error="error running Helm release deployment Step, failed to deploy helm release: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.37:15017, code 500: 500 Internal Server Error" time=2026-03-16T16:27:36.957Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-03-16T16:27:37.532Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy values="map[audit:map[connectSocket:false] azure:map[clientId:9b335788-9698-4c2e-b8b6-e774d43dd7a8 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] clustersService:map[namespace:clusters-service] cosmosDB:map[name:arohcpprow-rp-j8745984 url:https://arohcpprow-rp-j8745984.documents.azure.com:443/] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:admin-api-cert-prow-j8745984] deployment:map[limits:map[cpu:1 memory:1Gi] replicas:2 requests:map[cpu:200m memory:512Mi] zoneCount:4] fpa:map[certName:firstPartyCert2 clientId:b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 keyVaultName:aro-hcp-dev-svc-kv] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:admin-api-https] host:admin.j8745984.hcpsvc.osadev.cloud] image:map[digest:sha256:765bfaa0fea5572f7c547c3041dda88339289f0a850136bf92b4d880bb2d886f registry:arohcpsvcdev.azurecr.io repository:ci-op-yg9z9inx/pipeline] kusto:map[endpoint:https://hcp-dev-us-2.eastus2.kusto.windows.net] location:westus3 mise:map[deploy:false policyLabel:Geneva Actions] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-yg9z9inx/pipeline:pull workloadIdentityClientId:4e03dc43-0ff5-4d72-979f-67d7f7f56198 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:admin-api port:8443] serviceAccount:map[name:admin-api workloadIdentityClientId:163d854a-7dae-475f-9cc3-141cdae7b1c5 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] sessiongate:map[maxSessionTTL:24h minSessionTTL:1m mise:map[policyLabel:Session Gate] namespace:sessiongate servicePort:8080] virtualService:map[host:admin.j8745984.hcpsvc.osadev.cloud]]" time=2026-03-16T16:27:37.532Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy time=2026-03-16T16:27:37.627Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace aro-hcp-admin-api: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/aro-hcp-admin-api?fieldManager=templatize&force=false\\\": Post \\\"https://guard.69b82b1a32e3a9000188c805.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69b82b1a32e3a9000188c805.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (patch namespaces aro-hcp-admin-api)"
15:26:22	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head dcec1a0	n/a	provision	failed	#4393 (open)	1	provision (1) failed post-install: Unauthorized 15:26:22 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce · suite step graph Show raw failure time=2026-03-16T16:22:40.958Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce description="Step deploy-mce\n Kind: Helm\n" time=2026-03-16T16:22:41.564Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f imageRegistry:arohcpsvcdev.azurecr.io imageRootRepository:acm-d-cache]" time=2026-03-16T16:22:41.564Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:22:41.623Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce namespace=multicluster-engine time=2026-03-16T16:22:41.738Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce dryRun=false time=2026-03-16T16:22:41.738Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:22:41.855Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:24:41.768Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="failed post-install: Unauthorized" time=2026-03-16T16:24:41.768Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:24:41.768Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:24:45.935Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce release=mce namespace=multicluster-engine status=failed description="Release \"mce\" failed: failed post-install: Unauthorized" values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f imageRegistry:arohcpsvcdev.azurecr.io imageRootRepository:acm-d-cache]" time=2026-03-16T16:24:45.936Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/7SUS2/TQBSF9/4VV161kqdqXCcpQV3wWiFYACsQi/H4xJ56Xp1HIlf58chNQ6JKIRGC7Zmjz+Nz555MIZJHsMkLBLqjhkceea1wYet7iPhRmmYRopemLWgrfeYavyWsYJ6U4LjYyZc/MiKi/Cv8Sgq8EcImE/OCcp1UlEKlEOEZTCsNmHXwPFp/5DwvdjDhcRLC1qg7a/uzYdbBsJ1Jc8NbaJjIpOYtmEtKMeHRwETJVTiBfbdVv1iFt9I00rTH7utSp6ub+Wrq7KtqsJUzrmnn02nFZTn36lFX+jZNH7p53YyI3QdG8jFkWQ0uzfvlEMPgy0llpLSdR0r9bYle3yzXj76zKuDhGGKfztPczs2ahT/6X1A/WSOfp/3sU3IJMQgFFiKPYBrRSxHYqjw/7v+Q84kJ/ru438MpO4yv7m9XJPv5OutTDW8QET6MOxmyDa07eFCUGiFy7ahGXAOGLhoeMcoXeXldztj1DZvMvk1mi7JcVJPv+SVdXdFRT7X1XGYburfSUC9NcyeNgT9oEmtoXyCHzfGyMrINOW/H0/1FC3K22ZrPgxSkbFuQRgi8HZHWN/BUDwc/3yCIXwEAAP//5ifO5u8EAAA=" time=2026-03-16T16:24:45.937Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:24:45.938Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
15:20:06	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c6d34c6	n/a	provision	failed	⚠#4457 (open)	1	provision (1) make[2]: * [Makefile:106: deploy-cs-debug-jobs] Error 1 15:20:06 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy-cs-debug-jobs · suite step graph Show raw failure time=2026-03-16T16:26:33.542Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy-cs-debug-jobs description="Step deploy-cs-debug-jobs\n Kind: Shell\n Command: make deploy-cs-debug-jobs\n" time=2026-03-16T16:26:36.717Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy-cs-debug-jobs err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n./hack/helm.sh cs-debug-jobs helm-charts/cs-debug-jobs clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set deployDebugJobs=false\nRun Helm upgrade with release name cs-debug-jobs in namespace clusters-service\nRelease \"cs-debug-jobs\" does not exist. Installing it now.\nE0316 16:26:36.708578 2397 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: the server has asked for the client to provide credentials\" logger=\"UnhandledError\"\nError: could not get apiVersions from Kubernetes: could not get apiVersions from Kubernetes: the server has asked for the client to provide credentials\nmake[2]: * [Makefile:106: deploy-cs-debug-jobs] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
15:19:10	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 7b9532a · merge 7b9532a	n/a	provision	failed	#4455 (merged)	1	provision (1) failed post-install: Unauthorized 15:19:10 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce · suite step graph Show raw failure time=2026-03-16T16:21:34.726Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce description="Step deploy-mce\n Kind: Helm\n" time=2026-03-16T16:21:35.583Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f imageRegistry:arohcpsvcdev.azurecr.io imageRootRepository:acm-d-cache]" time=2026-03-16T16:21:35.583Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:21:35.646Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce namespace=multicluster-engine time=2026-03-16T16:21:35.754Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce dryRun=false time=2026-03-16T16:21:35.754Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:21:35.861Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:23:37.789Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="failed post-install: Unauthorized" time=2026-03-16T16:23:37.789Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:23:37.789Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:23:39.303Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="failed to get status for release mce: kubernetes cluster unreachable: the server has asked for the client to provide credentials" time=2026-03-16T16:23:39.303Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-03-16T16:23:39.303Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
14:57:41	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 3b95084 · merge 205984e	n/a	unknown	failed	#4449 (merged)	1	non-artifact-backed failure (no junit artifacts)
14:56:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 3d3fe8e · merge a0ece52	n/a	provision	failed	⚠#4481 (merged)	1	provision (1) failed to apply namespace multicluster-engine: an error on the server ("Internal Server Error: \\"/api/v1/namespaces/multicluster-engine?fieldManager=templatize&force=false\\": Post \\"<url>": proxy error from localhost:9443 while dialing 14:56:14 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph Show raw failure time=2026-03-16T16:24:00.786Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-03-16T16:24:01.400Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-03-16T16:24:01.400Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-03-16T16:24:01.504Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace multicluster-engine: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/multicluster-engine?fieldManager=templatize&force=false\\\": Post \\\"https://guard.69b82a8e54541c000131182e.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.69b82a8e54541c000131182e.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (patch namespaces multicluster-engine)"
14:49:49	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head f0eb454 · merge f0eb454	n/a	unknown	failed	#4241 (merged)	1	non-artifact-backed failure (no junit artifacts)
14:47:17	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 3f58874 · merge 138964d	n/a	unknown	failed	#4226 (merged)	1	non-artifact-backed failure (no junit artifacts)
14:28:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 89dbea9 · merge 0d2c310	n/a	provision	failed	⚠#4224 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code DeploymentScriptOperationFailed; detail message The container group '<id>' is still transitioning, please retry later. 14:28:07 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-03-16T15:55:01.252Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-03-16T15:55:03.096Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-03-16T15:55:04.692Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=7a202efbc0acb584cd9967ea2396ce403b176cb14a71bf6db367a49493f4a335 time=2026-03-16T16:09:08.071Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j6228224-svc/providers/Microsoft.Resources/deployments/7a202efbc0acb584cd9967ea2396ce403b176cb14a71bf6db367a49493f4a335/operationStatuses/08584279303816608430\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/frontend-cert-zqyt3pxcol57e\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deploymentScripts/newCert-frontend-cert-prow-j6228224\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentScriptOperationFailed\\\",\\r\\n \\\"message\\\": \\\"The container group 'dbon63o3dfre6azscripts' is still transitioning, please retry later.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
14:10:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 421438d · merge 9baa329	n/a	e2e	failed	⚠#4461 (merged)	4	e2e (4) missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 14:10:55 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000d776a0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000d776a0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 14:10:55 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error: <errors.joinError \| 0xc0012701e0>: failed to cleanup resource group: failed listing hcp clusters in resourcegroup="basic-cluster-j2w6hk": GET https://rp.j3045376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-j2w6hk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <fmt.wrapError \| 0xc0009f48c0>{ msg: "failed to cleanup resource group: failed listing hcp clusters in resourcegroup=\"basic-cluster-j2w6hk\": GET https://rp.j3045376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-j2w6hk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0009f4860>{ msg: "failed listing hcp clusters in resourcegroup=\"basic-cluster-j2w6hk\": GET https://rp.j3045376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-j2w6hk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001102540>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["2669"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "ff126f3d-baf1-4954-a9ec-cc35a45b7494", ], "Date": [ "Mon, 16 Mar 2026 16:32:56 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc0009f46c0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error: <errors.joinError \| 0xc0012701e0>: failed to cleanup resource group: failed listing hcp clusters in resourcegroup="basic-cluster-j2w6hk": GET https://rp.j3045376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-j2w6hk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <fmt.wrapError \| 0xc0009f48c0>{ msg: "failed to cleanup resource group: failed listing hcp clusters in resourcegroup=\"basic-cluster-j2w6hk\": GET https://rp.j3045376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-j2w6hk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0009f4860>{ msg: "failed listing hcp clusters in resourcegroup=\"basic-cluster-j2w6hk\": GET https://rp.j3045376.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-j2w6hk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001102540>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["2669"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "ff126f3d-baf1-4954-a9ec-cc35a45b7494", ], "Date": [ "Mon, 16 Mar 2026 16:32:56 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc0009f46c0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Error 14:10:55 UTC · failed at e2e · test Engineering should be able to retrieve expected metrics from the /metrics endpoint · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward Unexpected error: <fmt.wrapError \| 0xc000a026c0>: port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.9:10250, code 500: 500 Internal Server Error { msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.9:10250, code 500: 500 Internal Server Error", err: <errors.errorString \| 0xc00094c780>{ s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.9:10250, code 500: 500 Internal Server Error", }, } occurred fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward Unexpected error: <fmt.wrapError \| 0xc000a026c0>: port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.9:10250, code 500: 500 Internal Server Error { msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.9:10250, code 500: 500 Internal Server Error", err: <errors.errorString \| 0xc00094c780>{ s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.9:10250, code 500: 500 Internal Server Error", }, } occurred ERROR CODE: ResourceCollectionRequestsThrottled; provider Microsoft.Network 14:10:55 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error: <errors.joinError \| 0xc00156c168>: failed to cleanup resource group: failed to detach NSGs from subnets in resource group external-auth-cluster-27vjn9: failed listing vnets in resource group external-auth-cluster-27vjn9: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-27vjn9/providers/Microsoft.Network/virtualNetworks -------------------------------------------------------------------------------- RESPONSE 429: 429 Too Many Requests ERROR CODE: ResourceCollectionRequestsThrottled -------------------------------------------------------------------------------- { "error": { "code": "ResourceCollectionRequestsThrottled", "message": "Operation 'Microsoft.Network/virtualNetworks/read' failed as server encountered too many requests. Please try after '40' seconds. Tracking Id is 'b2596673-aadc-4155-b729-25f1c0c9d169'." } } -------------------------------------------------------------------------------- { errs: [ <fmt.wrapError \| 0xc000f08420>{ msg: "failed to cleanup resource group: failed to detach NSGs from subnets in resource group external-auth-cluster-27vjn9: failed listing vnets in resource group external-auth-cluster-27vjn9: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-27vjn9/providers/Microsoft.Network/virtualNetworks\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: ResourceCollectionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceCollectionRequestsThrottled\",\n \"message\": \"Operation 'Microsoft.Network/virtualNetworks/read' failed as server encountered too many requests. Please try after '40' seconds. Tracking Id is 'b2596673-aadc-4155-b729-25f1c0c9d169'.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f08400>{ msg: "failed to detach NSGs from subnets in resource group external-auth-cluster-27vjn9: failed listing vnets in resource group external-auth-cluster-27vjn9: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-27vjn9/providers/Microsoft.Network/virtualNetworks\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: ResourceCollectionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceCollectionRequestsThrottled\",\n \"message\": \"Operation 'Microsoft.Network/virtualNetworks/read' failed as server encountered too many requests. Please try after '40' seconds. Tracking Id is 'b2596673-aadc-4155-b729-25f1c0c9d169'.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f083c0>{ msg: "failed listing vnets in resource group external-auth-cluster-27vjn9: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-27vjn9/providers/Microsoft.Network/virtualNetworks\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: ResourceCollectionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceCollectionRequestsThrottled\",\n \"message\": \"Operation 'Microsoft.Network/virtualNetworks/read' failed as server encountered too many requests. Please try after '40' seconds. Tracking Id is 'b2596673-aadc-4155-b729-25f1c0c9d169'.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000a88750>{ ErrorCode: "ResourceCollectionRequestsThrottled", StatusCode: 429, RawResponse: { Status: "429 Too Many Requests", StatusCode: 429, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Cache": ["CONFIG_NOCACHE"], "Pragma": ["no-cache"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error: <errors.joinError \| 0xc00156c168>: failed to cleanup resource group: failed to detach NSGs from subnets in resource group external-auth-cluster-27vjn9: failed listing vnets in resource group external-auth-cluster-27vjn9: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-27vjn9/providers/Microsoft.Network/virtualNetworks -------------------------------------------------------------------------------- RESPONSE 429: 429 Too Many Requests ERROR CODE: ResourceCollectionRequestsThrottled -------------------------------------------------------------------------------- { "error": { "code": "ResourceCollectionRequestsThrottled", "message": "Operation 'Microsoft.Network/virtualNetworks/read' failed as server encountered too many requests. Please try after '40' seconds. Tracking Id is 'b2596673-aadc-4155-b729-25f1c0c9d169'." } } -------------------------------------------------------------------------------- { errs: [ <fmt.wrapError \| 0xc000f08420>{ msg: "failed to cleanup resource group: failed to detach NSGs from subnets in resource group external-auth-cluster-27vjn9: failed listing vnets in resource group external-auth-cluster-27vjn9: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-27vjn9/providers/Microsoft.Network/virtualNetworks\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: ResourceCollectionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceCollectionRequestsThrottled\",\n \"message\": \"Operation 'Microsoft.Network/virtualNetworks/read' failed as server encountered too many requests. Please try after '40' seconds. Tracking Id is 'b2596673-aadc-4155-b729-25f1c0c9d169'.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f08400>{ msg: "failed to detach NSGs from subnets in resource group external-auth-cluster-27vjn9: failed listing vnets in resource group external-auth-cluster-27vjn9: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-27vjn9/providers/Microsoft.Network/virtualNetworks\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: ResourceCollectionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceCollectionRequestsThrottled\",\n \"message\": \"Operation 'Microsoft.Network/virtualNetworks/read' failed as server encountered too many requests. Please try after '40' seconds. Tracking Id is 'b2596673-aadc-4155-b729-25f1c0c9d169'.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f083c0>{ msg: "failed listing vnets in resource group external-auth-cluster-27vjn9: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-27vjn9/providers/Microsoft.Network/virtualNetworks\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: ResourceCollectionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"ResourceCollectionRequestsThrottled\",\n \"message\": \"Operation 'Microsoft.Network/virtualNetworks/read' failed as server encountered too many requests. Please try after '40' seconds. Tracking Id is 'b2596673-aadc-4155-b729-25f1c0c9d169'.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000a88750>{ ErrorCode: "ResourceCollectionRequestsThrottled", StatusCode: 429, RawResponse: { Status: "429 Too Many Requests", StatusCode: 429, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Cache": ["CONFIG_NOCACHE"], "Pragma": ["no-cache"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
13:19:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head f3b153a	n/a	unknown	failed	#1955 (open)	1	non-artifact-backed failure (no junit artifacts)
13:19:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 37c1fdb	n/a	unknown	failed	#2312 (open)	1	non-artifact-backed failure (no junit artifacts)
13:18:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 88f665f	n/a	unknown	failed	#2422 (open)	1	non-artifact-backed failure (no junit artifacts)
13:17:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c8ca820	n/a	unknown	failed	#2593 (open)	1	non-artifact-backed failure (no junit artifacts)
13:17:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6085e52	n/a	unknown	failed	#2615 (open)	1	non-artifact-backed failure (no junit artifacts)
13:16:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 860d47c	n/a	unknown	failed	#2620 (open)	1	non-artifact-backed failure (no junit artifacts)
13:15:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 421c402	n/a	unknown	failed	#2858 (open)	1	non-artifact-backed failure (no junit artifacts)
13:14:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d7f12c9	n/a	unknown	failed	#2873 (open)	1	non-artifact-backed failure (no junit artifacts)
13:13:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4a5b0be	n/a	unknown	failed	#3045 (open)	1	non-artifact-backed failure (no junit artifacts)
13:11:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head bed5752	n/a	unknown	failed	#3061 (open)	1	non-artifact-backed failure (no junit artifacts)
13:09:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 815b1f3	n/a	unknown	failed	#3129 (open)	1	non-artifact-backed failure (no junit artifacts)
13:09:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0e2cea3	n/a	unknown	failed	#3172 (closed)	1	non-artifact-backed failure (no junit artifacts)
13:08:44	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 056231a	n/a	unknown	failed	#4117 (closed)	1	non-artifact-backed failure (no junit artifacts)
13:07:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c682e6c	n/a	unknown	failed	#3243 (open)	1	non-artifact-backed failure (no junit artifacts)
13:06:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4bed8a1	n/a	unknown	failed	#3390 (open)	1	non-artifact-backed failure (no junit artifacts)
13:05:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 217ba23	n/a	unknown	failed	#3414 (open)	1	non-artifact-backed failure (no junit artifacts)
13:05:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c58bad3	n/a	unknown	failed	#3557 (open)	1	non-artifact-backed failure (no junit artifacts)
13:05:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9583434	n/a	unknown	failed	#3564 (open)	1	non-artifact-backed failure (no junit artifacts)
13:04:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c5298fc	n/a	unknown	failed	#3585 (open)	1	non-artifact-backed failure (no junit artifacts)
13:04:21	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2be65f9	n/a	unknown	failed	#4403 (open)	1	non-artifact-backed failure (no junit artifacts)
13:03:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head fc77837	n/a	provision	failed	#3596 (open)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 13:03:59 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-03-16T15:38:43.499Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-03-16T15:40:41.168Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=0abd6cee-971e-4831-bf65-52563f64495b && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=838dbe80-d244-413a-9241-6ad2025a235b && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj3116672.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj3116672.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j3116672 -n arohcp-prow-dbcs-j3116672 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3116672/providers/Microsoft.Network/dnszones/j3116672.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j3116672-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j3116672-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpprow \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set global.imageDigest=sha256:6ec7ecb11515ea6f8be560c4c043b518ef5c5ccc2a0aaafb2a45ee79c25a9d80 \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j3116672-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"232f5520-5f28-4117-986c-3714aa1e0fc7\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j3116672-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3116672-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j3116672-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
13:03:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 6029601 · merge 6029601	n/a	unknown	failed	#3661 (merged)	1	non-artifact-backed failure (no junit artifacts)
13:01:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7c52cdf	n/a	unknown	failed	#3680 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:58:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3900846	n/a	unknown	failed	#3696 (open)	1	non-artifact-backed failure (no junit artifacts)
12:57:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2b9a7c8	n/a	unknown	failed	#3704 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:56:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a81d500	n/a	unknown	failed	#3729 (open)	1	non-artifact-backed failure (no junit artifacts)
12:55:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 12f8f99	n/a	unknown	failed	#3760 (open)	1	non-artifact-backed failure (no junit artifacts)
12:54:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 15a2d96	n/a	unknown	failed	#3779 (open)	1	non-artifact-backed failure (no junit artifacts)
12:53:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 704ba86	n/a	unknown	failed	#3794 (open)	1	non-artifact-backed failure (no junit artifacts)
12:52:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 51248ea	n/a	unknown	failed	#3795 (open)	1	non-artifact-backed failure (no junit artifacts)
12:52:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d911993	n/a	unknown	failed	#3823 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:51:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4c4a644	n/a	unknown	failed	#3827 (open)	1	non-artifact-backed failure (no junit artifacts)
12:50:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c90f337	n/a	unknown	failed	#3839 (open)	1	non-artifact-backed failure (no junit artifacts)
12:49:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head bde613f	n/a	unknown	failed	#3840 (open)	1	non-artifact-backed failure (no junit artifacts)
12:49:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0a38030	n/a	unknown	failed	#3841 (open)	1	non-artifact-backed failure (no junit artifacts)
12:46:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head bbd7049	n/a	unknown	failed	#3860 (open)	1	non-artifact-backed failure (no junit artifacts)
12:43:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7b88114	n/a	unknown	failed	#3880 (open)	1	non-artifact-backed failure (no junit artifacts)
12:40:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head afc3425 · merge e9ca15d	n/a	unknown	failed	#3907 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:39:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2841bf7	n/a	unknown	failed	#3913 (open)	1	non-artifact-backed failure (no junit artifacts)
12:38:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head babd84c	n/a	unknown	failed	#3921 (open)	1	non-artifact-backed failure (no junit artifacts)
12:37:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ff393c2	n/a	provision	failed	#3925 (open)	1	provision (1) failed to apply namespace prometheus: Unauthorized 12:37:47 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite step graph Show raw failure time=2026-03-16T15:11:09.550Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-03-16T15:11:10.202Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpprow kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:17fc37d9fe70a0efa4fdaf609bd03299013faa0fee16b77815469eb37a879e7c]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:d7e8257f8d8bce64b6df55f81fba92011a6a77269b3350f8b997b152af348dba]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:b53be343389b63fa880639565d6067ba933f3f009642cd2e2c02bcdc189a63de] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:9f029892d072c33a88f7120170c900a3f2e6b9758f16477c5dec60c3ae7504e6]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:414103ed-9e0e-48b0-af39-d5334fe1b53c]] prometheusSpec:map[externalLabels:map[cluster:prow-j2281088-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:dca4b35f73046dc4803c0160c2e6be5a1736a716b26ef35590864d36dfdf950a] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j2281088-svc-yeyq.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-3ea31b0dcf98492e9a18d301fe28c4d7/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-03-16T15:11:10.202Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-03-16T15:11:10.846Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace prometheus: Unauthorized"
12:37:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head baff291	n/a	unknown	failed	#3952 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:36:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 23fa46c	n/a	unknown	failed	#3957 (open)	1	non-artifact-backed failure (no junit artifacts)
12:35:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head b65c3a6 · merge b65c3a6	n/a	unknown	failed	#3960 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:34:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head fee956d · merge 445d308	n/a	unknown	failed	#3973 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:33:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head dfad7bf	n/a	unknown	failed	#3975 (open)	1	non-artifact-backed failure (no junit artifacts)
12:33:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a61671e	n/a	unknown	failed	#3977 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:31:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head be9daed · merge 9feee09	n/a	unknown	failed	#4004 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:30:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9e94925	n/a	unknown	failed	#4040 (open)	1	non-artifact-backed failure (no junit artifacts)
12:29:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head f5b50ce	n/a	unknown	failed	#4044 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:28:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3207545	n/a	unknown	failed	#4045 (open)	1	non-artifact-backed failure (no junit artifacts)
12:28:30	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 3b95084 · merge 205984e	n/a	unknown	failed	#4449 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:28:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 456c77e	n/a	unknown	failed	#4046 (open)	1	non-artifact-backed failure (no junit artifacts)
12:27:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0185c7c	n/a	unknown	failed	#4069 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:25:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a35acbe	n/a	unknown	failed	#4084 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:24:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8f711fa	n/a	unknown	failed	#4095 (open)	1	non-artifact-backed failure (no junit artifacts)
12:23:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2eda048	n/a	unknown	failed	#4117 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:22:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 1c91345 · merge 4f10ef5	n/a	unknown	failed	#4133 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:21:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c004a2c	n/a	unknown	failed	#4137 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:21:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c179b3a	n/a	unknown	failed	#4138 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:20:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 730aedc	n/a	unknown	failed	#4157 (open)	1	non-artifact-backed failure (no junit artifacts)
12:19:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 661d925	n/a	unknown	failed	#4161 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:18:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6881981	n/a	unknown	failed	#4171 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:18:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 97b8851	n/a	unknown	failed	#4185 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:17:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head c45e5a9 · merge ad7003e	n/a	unknown	failed	#4188 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:16:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head cbb5a98	n/a	unknown	failed	#4189 (closed)	1	non-artifact-backed failure (no junit artifacts)
12:15:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 83e208a	n/a	unknown	failed	#4192 (open)	1	non-artifact-backed failure (no junit artifacts)
12:15:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6bb6ada	n/a	unknown	failed	#4193 (open)	1	non-artifact-backed failure (no junit artifacts)
12:14:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1b8d093	n/a	unknown	failed	#4198 (open)	1	non-artifact-backed failure (no junit artifacts)
12:13:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head bafae7a	n/a	e2e	failed	★#4210 (open)	4	e2e (4) missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 12:13:11 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc0015727f0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc0015727f0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 12:13:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapError \| 0xc001546b80>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-kpmzkj" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006", "name": "8a32d5f3-83b0-45cf-974c-a52cbc080006", "status": "Failed", "startTime": "2026-03-16T14:54:48.303502861Z", "endTime": "2026-03-16T14:55:48.444999538Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kpmzkj\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006\",\n \"name\": \"8a32d5f3-83b0-45cf-974c-a52cbc080006\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:48.303502861Z\",\n \"endTime\": \"2026-03-16T14:55:48.444999538Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001546b60>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kpmzkj\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006\",\n \"name\": \"8a32d5f3-83b0-45cf-974c-a52cbc080006\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:48.303502861Z\",\n \"endTime\": \"2026-03-16T14:55:48.444999538Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapError \| 0xc001546b80>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-kpmzkj" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006", "name": "8a32d5f3-83b0-45cf-974c-a52cbc080006", "status": "Failed", "startTime": "2026-03-16T14:54:48.303502861Z", "endTime": "2026-03-16T14:55:48.444999538Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kpmzkj\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006\",\n \"name\": \"8a32d5f3-83b0-45cf-974c-a52cbc080006\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:48.303502861Z\",\n \"endTime\": \"2026-03-16T14:55:48.444999538Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001546b60>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kpmzkj\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a32d5f3-83b0-45cf-974c-a52cbc080006\",\n \"name\": \"8a32d5f3-83b0-45cf-974c-a52cbc080006\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:48.303502861Z\",\n \"endTime\": \"2026-03-16T14:55:48.444999538Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-150/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 12:13:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc001540120>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-f5njwd" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab", "name": "01e17f5d-e2ff-4f99-9918-11e18c2490ab", "status": "Failed", "startTime": "2026-03-16T14:54:56.798653664Z", "endTime": "2026-03-16T14:55:38.486152327Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-f5njwd\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab\",\n \"name\": \"01e17f5d-e2ff-4f99-9918-11e18c2490ab\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:56.798653664Z\",\n \"endTime\": \"2026-03-16T14:55:38.486152327Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001540100>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-f5njwd\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab\",\n \"name\": \"01e17f5d-e2ff-4f99-9918-11e18c2490ab\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:56.798653664Z\",\n \"endTime\": \"2026-03-16T14:55:38.486152327Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionReque... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc001540120>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-f5njwd" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab", "name": "01e17f5d-e2ff-4f99-9918-11e18c2490ab", "status": "Failed", "startTime": "2026-03-16T14:54:56.798653664Z", "endTime": "2026-03-16T14:55:38.486152327Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-f5njwd\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab\",\n \"name\": \"01e17f5d-e2ff-4f99-9918-11e18c2490ab\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:56.798653664Z\",\n \"endTime\": \"2026-03-16T14:55:38.486152327Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001540100>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-f5njwd\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e17f5d-e2ff-4f99-9918-11e18c2490ab\",\n \"name\": \"01e17f5d-e2ff-4f99-9918-11e18c2490ab\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:56.798653664Z\",\n \"endTime\": \"2026-03-16T14:55:38.486152327Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-140/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionReque... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 12:13:11 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000f407e0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-kmh24k" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08", "name": "fcc9c987-dc72-47a5-acd2-7492bcbbcd08", "status": "Failed", "startTime": "2026-03-16T14:54:48.473384961Z", "endTime": "2026-03-16T14:56:32.581499648Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-kmh24k\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08\",\n \"name\": \"fcc9c987-dc72-47a5-acd2-7492bcbbcd08\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:48.473384961Z\",\n \"endTime\": \"2026-03-16T14:56:32.581499648Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f407c0>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-kmh24k\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08\",\n \"name\": \"fcc9c987-dc72-47a5-acd2-7492bcbbcd08\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:48.473384961Z\",\n \"endTime\": \"2026-03-16T14:56:32.581499648Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000f407e0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-kmh24k" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08", "name": "fcc9c987-dc72-47a5-acd2-7492bcbbcd08", "status": "Failed", "startTime": "2026-03-16T14:54:48.473384961Z", "endTime": "2026-03-16T14:56:32.581499648Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-kmh24k\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08\",\n \"name\": \"fcc9c987-dc72-47a5-acd2-7492bcbbcd08\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:48.473384961Z\",\n \"endTime\": \"2026-03-16T14:56:32.581499648Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f407c0>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-kmh24k\" to finish creating: GET https://rp.j7942272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fcc9c987-dc72-47a5-acd2-7492bcbbcd08\",\n \"name\": \"fcc9c987-dc72-47a5-acd2-7492bcbbcd08\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:54:48.473384961Z\",\n \"endTime\": \"2026-03-16T14:56:32.581499648Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-202/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
12:12:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 39e7850	n/a	unknown	failed	#4211 (open)	1	non-artifact-backed failure (no junit artifacts)
12:11:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head c8926b8 · merge 8550ece	n/a	unknown	failed	#4219 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:10:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 89dbea9 · merge 0d2c310	n/a	unknown	failed	#4224 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:09:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 3f58874 · merge 138964d	n/a	unknown	failed	#4226 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:06:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 9d6e88a · merge ac3354a	n/a	unknown	failed	#4234 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:06:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head fea5b77 · merge fea5b77	n/a	unknown	failed	#4235 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:06:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head f0eb454 · merge f0eb454	n/a	unknown	failed	#4241 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:05:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7ec0c2c	n/a	unknown	failed	#4255 (open)	1	non-artifact-backed failure (no junit artifacts)
12:04:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 77f6ef2 · merge f9b0dd4	n/a	unknown	failed	#4257 (merged)	1	non-artifact-backed failure (no junit artifacts)
12:03:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 66a5158	n/a	unknown	failed	#4272 (open)	1	non-artifact-backed failure (no junit artifacts)
12:01:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head f55caa5	n/a	unknown	failed	#4281 (open)	1	non-artifact-backed failure (no junit artifacts)
12:00:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 39e398e	n/a	unknown	failed	#4286 (open)	1	non-artifact-backed failure (no junit artifacts)
12:00:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 932223a	n/a	unknown	failed	#4291 (closed)	1	non-artifact-backed failure (no junit artifacts)
11:59:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c0fed25	n/a	unknown	failed	#4305 (closed)	1	non-artifact-backed failure (no junit artifacts)
11:55:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4c2e431	n/a	e2e	failed	★#4342 (closed)	16	e2e (16) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e80240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-nqc97l" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00110e480>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-nqc97l\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00110e460>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-nqc97l\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00110e440>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-nqc97l\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e80240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-nqc97l" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00110e480>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-nqc97l\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00110e460>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-nqc97l\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00110e440>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-nqc97l\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0012200d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m4snht" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00075fc80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m4snht\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00075fc60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m4snht\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00075fc40>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m4snht\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0012200d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m4snht" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00075fc80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m4snht\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00075fc60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m4snht\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00075fc40>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m4snht\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00161e1f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-jqdc5f" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e82fa0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jqdc5f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e82f80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jqdc5f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e82f60>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jqdc5f\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00161e1f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-jqdc5f" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e82fa0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jqdc5f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e82f80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jqdc5f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e82f60>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jqdc5f\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00021a4c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-ksdsq5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007a57e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-ksdsq5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007a57c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-ksdsq5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007a57a0>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-ksdsq5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00021a4c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-ksdsq5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007a57e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-ksdsq5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007a57c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-ksdsq5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007a57a0>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-ksdsq5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f481c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mzs9x9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a776e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-mzs9x9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a776c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-mzs9x9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a776a0>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-mzs9x9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f481c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mzs9x9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a776e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-mzs9x9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a776c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-mzs9x9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a776a0>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-mzs9x9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:55:35 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc00093f6a0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc00093f6a0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001774210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-tcncf7" in resourcegroup="rg-zstream-upgrade-4-21-zs86xz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d413e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-tcncf7\" in resourcegroup=\"rg-zstream-upgrade-4-21-zs86xz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d413c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-tcncf7\" in resourcegroup=\"rg-zstream-upgrade-4-21-zs86xz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d413a0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-tcncf7\" in resourcegroup=\"rg-zstream-upgrade-4-21-zs86xz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001774210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-tcncf7" in resourcegroup="rg-zstream-upgrade-4-21-zs86xz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d413e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-tcncf7\" in resourcegroup=\"rg-zstream-upgrade-4-21-zs86xz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d413c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-tcncf7\" in resourcegroup=\"rg-zstream-upgrade-4-21-zs86xz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d413a0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-tcncf7\" in resourcegroup=\"rg-zstream-upgrade-4-21-zs86xz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift 11:55:35 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapError \| 0xc000a0fee0>: failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-tm8z9r" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8", "name": "276ecb68-8f36-4788-a27a-b959fd469eb8", "status": "Failed", "startTime": "2026-03-16T14:30:45.201431526Z", "endTime": "2026-03-16T14:31:35.067401014Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-tm8z9r/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tm8z9r\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\",\n \"name\": \"276ecb68-8f36-4788-a27a-b959fd469eb8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:30:45.201431526Z\",\n \"endTime\": \"2026-03-16T14:31:35.067401014Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-tm8z9r/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a0fec0>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tm8z9r\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\n-----------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapError \| 0xc000a0fee0>: failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-tm8z9r" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8", "name": "276ecb68-8f36-4788-a27a-b959fd469eb8", "status": "Failed", "startTime": "2026-03-16T14:30:45.201431526Z", "endTime": "2026-03-16T14:31:35.067401014Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-tm8z9r/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tm8z9r\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\",\n \"name\": \"276ecb68-8f36-4788-a27a-b959fd469eb8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:30:45.201431526Z\",\n \"endTime\": \"2026-03-16T14:31:35.067401014Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-tm8z9r/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-131/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a0fec0>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-tm8z9r\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276ecb68-8f36-4788-a27a-b959fd469eb8\n-----------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fd6228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-g9db66" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f87640>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-g9db66\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f87600>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-g9db66\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f875e0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-g9db66\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fd6228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-g9db66" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f87640>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-g9db66\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f87600>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-g9db66\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f875e0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-g9db66\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001590108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-9q9dbq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006e8e20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-9q9dbq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006e8e00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-9q9dbq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006e8de0>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-9q9dbq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001590108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-9q9dbq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006e8e20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-9q9dbq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006e8e00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-9q9dbq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006e8de0>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-9q9dbq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0007d0228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-mlbqd7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000808660>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-mlbqd7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000808640>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-mlbqd7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000808620>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-mlbqd7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0007d0228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-mlbqd7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000808660>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-mlbqd7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000808640>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-mlbqd7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000808620>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-mlbqd7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001780120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-rxbrqv" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001423540>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rxbrqv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001423520>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rxbrqv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001423500>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rxbrqv\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001780120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-rxbrqv" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001423540>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rxbrqv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001423520>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rxbrqv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001423500>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rxbrqv\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f4c2e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-rflvdb" in resourcegroup="admin-credential-lifecycle-test-rx8cq2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e5aec0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-rflvdb\" in resourcegroup=\"admin-credential-lifecycle-test-rx8cq2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e5aea0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-rflvdb\" in resourcegroup=\"admin-credential-lifecycle-test-rx8cq2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e5ae80>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-rflvdb\" in resourcegroup=\"admin-credential-lifecycle-test-rx8cq2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f4c2e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-rflvdb" in resourcegroup="admin-credential-lifecycle-test-rx8cq2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e5aec0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-rflvdb\" in resourcegroup=\"admin-credential-lifecycle-test-rx8cq2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e5aea0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-rflvdb\" in resourcegroup=\"admin-credential-lifecycle-test-rx8cq2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e5ae80>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-rflvdb\" in resourcegroup=\"admin-credential-lifecycle-test-rx8cq2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:55:35 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc001264100>: failed to create HCP cluster cluster-zstream-4-20-2h67vc: failed waiting for cluster="cluster-zstream-4-20-2h67vc" in resourcegroup="rg-zstream-upgrade-4-20-jmj74s" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c", "name": "5591d220-8f1b-4485-99fc-95c50a90e42c", "status": "Failed", "startTime": "2026-03-16T15:00:20.941704343Z", "endTime": "2026-03-16T15:01:07.240406639Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-2h67vc: failed waiting for cluster=\"cluster-zstream-4-20-2h67vc\" in resourcegroup=\"rg-zstream-upgrade-4-20-jmj74s\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c\",\n \"name\": \"5591d220-8f1b-4485-99fc-95c50a90e42c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T15:00:20.941704343Z\",\n \"endTime\": \"2026-03-16T15:01:07.240406639Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0012640e0>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-2h67vc\" in resourcegroup=\"rg-zstream-upgrade-4-20-jmj74s\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c\",\n \"name\": \"5591d220-8f1b-4485-99fc-95c50a90e42c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T15:00:20.941704343Z\",\n \"endTime\": \"2026-03-16T15:01:07.240406639Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc001264100>: failed to create HCP cluster cluster-zstream-4-20-2h67vc: failed waiting for cluster="cluster-zstream-4-20-2h67vc" in resourcegroup="rg-zstream-upgrade-4-20-jmj74s" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c", "name": "5591d220-8f1b-4485-99fc-95c50a90e42c", "status": "Failed", "startTime": "2026-03-16T15:00:20.941704343Z", "endTime": "2026-03-16T15:01:07.240406639Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-2h67vc: failed waiting for cluster=\"cluster-zstream-4-20-2h67vc\" in resourcegroup=\"rg-zstream-upgrade-4-20-jmj74s\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c\",\n \"name\": \"5591d220-8f1b-4485-99fc-95c50a90e42c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T15:00:20.941704343Z\",\n \"endTime\": \"2026-03-16T15:01:07.240406639Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0012640e0>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-2h67vc\" in resourcegroup=\"rg-zstream-upgrade-4-20-jmj74s\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5591d220-8f1b-4485-99fc-95c50a90e42c\",\n \"name\": \"5591d220-8f1b-4485-99fc-95c50a90e42c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T15:00:20.941704343Z\",\n \"endTime\": \"2026-03-16T15:01:07.240406639Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-65/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:55:35 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ffe2e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-wk26tz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00069fc80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-wk26tz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00069fc60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-wk26tz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00069fc20>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-wk26tz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ffe2e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-wk26tz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00069fc80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-wk26tz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00069fc60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-wk26tz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00069fc20>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-wk26tz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09d0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:55:35 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc0004930a0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-hf9qjl" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b", "name": "21c9583a-7160-4b35-a5ee-f381fc0ba33b", "status": "Failed", "startTime": "2026-03-16T14:55:06.527348633Z", "endTime": "2026-03-16T14:55:50.172982114Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-hf9qjl\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b\",\n \"name\": \"21c9583a-7160-4b35-a5ee-f381fc0ba33b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:55:06.527348633Z\",\n \"endTime\": \"2026-03-16T14:55:50.172982114Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000493060>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-hf9qjl\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b\",\n \"name\": \"21c9583a-7160-4b35-a5ee-f381fc0ba33b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:55:06.527348633Z\",\n \"endTime\": \"2026-03-16T14:55:50.172982114Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc0004930a0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-hf9qjl" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b", "name": "21c9583a-7160-4b35-a5ee-f381fc0ba33b", "status": "Failed", "startTime": "2026-03-16T14:55:06.527348633Z", "endTime": "2026-03-16T14:55:50.172982114Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-hf9qjl\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b\",\n \"name\": \"21c9583a-7160-4b35-a5ee-f381fc0ba33b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:55:06.527348633Z\",\n \"endTime\": \"2026-03-16T14:55:50.172982114Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000493060>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-hf9qjl\" to finish creating: GET https://rp.j4385408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21c9583a-7160-4b35-a5ee-f381fc0ba33b\",\n \"name\": \"21c9583a-7160-4b35-a5ee-f381fc0ba33b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:55:06.527348633Z\",\n \"endTime\": \"2026-03-16T14:55:50.172982114Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-233/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
11:54:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7f9a7eb	n/a	unknown	failed	#4343 (closed)	1	non-artifact-backed failure (no junit artifacts)
11:53:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 73b7fff	n/a	provision	failed	#4344 (open)	1	provision (1) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError("HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 11:53:59 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.HypershiftOperator/global/mirror-image · suite step graph Show raw failure time=2026-03-16T14:23:17.605Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.HypershiftOperator resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> hypershift.image.registry <nil>}:{<nil> hypershift.image.repository <nil>}@{<nil> hypershift.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-03-16T14:23:41.134Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.HypershiftOperator resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nFailed to log in to ACR arohcpsvcdev: ERROR: The command failed with an unexpected error. Here is the traceback:\nERROR: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 204, in _new_conn\n sock = connection.create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/connection.py\", line 60, in create_connection\n for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/python3.12/socket.py\", line 963, in getaddrinfo\n for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 787, in urlopen\n response = self._make_request(\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 488, in _make_request\n raise new_e\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 464, in _make_request\n self._validate_conn(conn)\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 1093, in _validate_conn\n conn.connect()\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 759, in connect\n self.sock = sock = self._new_conn()\n ^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 211, in _new_conn\n raise NameResolutionError(self.host, self, e) from e\nurllib3.exceptions.NameResolutionError: HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n resp = conn.urlopen(\n ^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 841, in urlopen\n retries = retries.increment(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/retry.py\", line 535, in increment\n raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"/usr/lib64/az/lib/python3.12/site-packages/knack/cli.py\", line 233, in invoke\n cmd_result = self.invocation.execute(args)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 682, in execute\n raise ex\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 812, in _run_jobs_serially\n results.append(self._run_job(expanded_arg, cmd_copy))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 781, in _run_job\n result = cmd_copy(params)\n ^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 336, in __call__\n return self.handler(args, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py\", line 120, in handler\n return op(command_args)\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/custom.py\", line 294, in acr_login\n login_server, _, password = get_login_credentials(\n ^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 505, in get_login_credentials\n return _get_credentials(cmd,\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 449, in _get_credentials\n return login_server, EMPTY_GUID, _get_aad_token(cli_ctx,\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 285, in _get_aad_token\n return _get_aad_token_after_challenge(cli_ctx,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 159, in _get_aad_token_after_challenge\n response = requests.post(url=authhost, data=urlencode(content), headers=headers,\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 115, in post\n return request(\"post\", url, data=data, json=json, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n return session.request(method=method, url=url, kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n resp = self.send(prep, send_kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n r = adapter.send(request, *kwargs)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 700, in send\n raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTo check existing issues, please visit: https://github.com/Azure/azure-cli/issues\n exit status 1"
11:53:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head d7ca9a9	n/a	e2e	failed	★#4346 (closed)	19	e2e (19) ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:53:11 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc0014d62c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-pp66nd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pp66nd/operationStatuses/08584279348069236799 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-2mq2n4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-pp66nd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pp66nd/operationStatuses/08584279348069236799\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-2mq2n4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc0014d62c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-pp66nd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pp66nd/operationStatuses/08584279348069236799 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-2mq2n4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-pp66nd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pp66nd/operationStatuses/08584279348069236799\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-2mq2n4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016b81e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-22nlpz" in resourcegroup="admin-credential-lifecycle-test-69cfx2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009d13a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-22nlpz\" in resourcegroup=\"admin-credential-lifecycle-test-69cfx2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009d1360>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-22nlpz\" in resourcegroup=\"admin-credential-lifecycle-test-69cfx2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009d1340>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-22nlpz\" in resourcegroup=\"admin-credential-lifecycle-test-69cfx2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016b81e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-22nlpz" in resourcegroup="admin-credential-lifecycle-test-69cfx2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009d13a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-22nlpz\" in resourcegroup=\"admin-credential-lifecycle-test-69cfx2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009d1360>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-22nlpz\" in resourcegroup=\"admin-credential-lifecycle-test-69cfx2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009d1340>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-22nlpz\" in resourcegroup=\"admin-credential-lifecycle-test-69cfx2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000b7c750>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-19-sxbwlr" in resourcegroup="rg-zstream-upgrade-4-19-gvfq6z" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00079ca00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-sxbwlr\" in resourcegroup=\"rg-zstream-upgrade-4-19-gvfq6z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00079c9e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-sxbwlr\" in resourcegroup=\"rg-zstream-upgrade-4-19-gvfq6z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00079c9c0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-sxbwlr\" in resourcegroup=\"rg-zstream-upgrade-4-19-gvfq6z\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000b7c750>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-19-sxbwlr" in resourcegroup="rg-zstream-upgrade-4-19-gvfq6z" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00079ca00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-sxbwlr\" in resourcegroup=\"rg-zstream-upgrade-4-19-gvfq6z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00079c9e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-sxbwlr\" in resourcegroup=\"rg-zstream-upgrade-4-19-gvfq6z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00079c9c0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-sxbwlr\" in resourcegroup=\"rg-zstream-upgrade-4-19-gvfq6z\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0015b60f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-dvpw9w" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e06ce0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-dvpw9w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e06cc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-dvpw9w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e06ca0>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-dvpw9w\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0015b60f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-dvpw9w" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e06ce0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-dvpw9w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e06cc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-dvpw9w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e06ca0>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-dvpw9w\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e10240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-fm7r92" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0016e0000>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-fm7r92\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b5bf60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-fm7r92\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b5bf40>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-fm7r92\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e10240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-fm7r92" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0016e0000>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-fm7r92\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b5bf60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-fm7r92\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b5bf40>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-fm7r92\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00108c2a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-47z72f" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a95100>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-47z72f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a950e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-47z72f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a950c0>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-47z72f\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00108c2a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-47z72f" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a95100>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-47z72f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a950e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-47z72f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a950c0>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-47z72f\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:53:11 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000a282a0>: failed to create HCP cluster cluster-zstream-4-21-pbt42n: failed waiting for cluster="cluster-zstream-4-21-pbt42n" in resourcegroup="rg-zstream-upgrade-4-21-9lv5nh" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8", "name": "dc08e50a-b558-47b3-9bce-e9c0b75671b8", "status": "Failed", "startTime": "2026-03-16T14:38:41.910090205Z", "endTime": "2026-03-16T14:39:43.777378228Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-pbt42n: failed waiting for cluster=\"cluster-zstream-4-21-pbt42n\" in resourcegroup=\"rg-zstream-upgrade-4-21-9lv5nh\" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8\",\n \"name\": \"dc08e50a-b558-47b3-9bce-e9c0b75671b8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:38:41.910090205Z\",\n \"endTime\": \"2026-03-16T14:39:43.777378228Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a28260>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-pbt42n\" in resourcegroup=\"rg-zstream-upgrade-4-21-9lv5nh\" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8\",\n \"name\": \"dc08e50a-b558-47b3-9bce-e9c0b75671b8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:38:41.910090205Z\",\n \"endTime\": \"2026-03-16T14:39:43.777378228Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subsc... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000a282a0>: failed to create HCP cluster cluster-zstream-4-21-pbt42n: failed waiting for cluster="cluster-zstream-4-21-pbt42n" in resourcegroup="rg-zstream-upgrade-4-21-9lv5nh" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8", "name": "dc08e50a-b558-47b3-9bce-e9c0b75671b8", "status": "Failed", "startTime": "2026-03-16T14:38:41.910090205Z", "endTime": "2026-03-16T14:39:43.777378228Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-pbt42n: failed waiting for cluster=\"cluster-zstream-4-21-pbt42n\" in resourcegroup=\"rg-zstream-upgrade-4-21-9lv5nh\" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8\",\n \"name\": \"dc08e50a-b558-47b3-9bce-e9c0b75671b8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:38:41.910090205Z\",\n \"endTime\": \"2026-03-16T14:39:43.777378228Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a28260>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-pbt42n\" in resourcegroup=\"rg-zstream-upgrade-4-21-9lv5nh\" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc08e50a-b558-47b3-9bce-e9c0b75671b8\",\n \"name\": \"dc08e50a-b558-47b3-9bce-e9c0b75671b8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:38:41.910090205Z\",\n \"endTime\": \"2026-03-16T14:39:43.777378228Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-203/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subsc... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e24690>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-w4wcl2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000c97ec0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-w4wcl2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c97e80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-w4wcl2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c97e60>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-w4wcl2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e24690>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-w4wcl2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000c97ec0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-w4wcl2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c97e80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-w4wcl2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c97e60>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-w4wcl2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] 11:53:11 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.002s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Expected <[]string \| len:1, cap:1>: [ "image-registry (Available=False)", ] to be empty fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.002s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Expected <[]string \| len:1, cap:1>: [ "image-registry (Available=False)", ] to be empty failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0015860f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-jgrbzg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f27ac0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-jgrbzg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f27aa0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-jgrbzg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f27a80>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-jgrbzg\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0015860f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-jgrbzg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f27ac0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-jgrbzg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f27aa0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-jgrbzg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f27a80>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-jgrbzg\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:53:11 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000a741e0>: failed to create HCP cluster cluster-zstream-4-20-8kc9g7: failed waiting for cluster="cluster-zstream-4-20-8kc9g7" in resourcegroup="rg-zstream-upgrade-4-20-dcjhgg" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f", "name": "724b22a8-140c-4634-bdee-1da54bd44a5f", "status": "Failed", "startTime": "2026-03-16T14:36:41.379585924Z", "endTime": "2026-03-16T14:37:40.220183063Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-8kc9g7: failed waiting for cluster=\"cluster-zstream-4-20-8kc9g7\" in resourcegroup=\"rg-zstream-upgrade-4-20-dcjhgg\" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f\",\n \"name\": \"724b22a8-140c-4634-bdee-1da54bd44a5f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:41.379585924Z\",\n \"endTime\": \"2026-03-16T14:37:40.220183063Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a74180>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-8kc9g7\" in resourcegroup=\"rg-zstream-upgrade-4-20-dcjhgg\" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f\",\n \"name\": \"724b22a8-140c-4634-bdee-1da54bd44a5f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:41.379585924Z\",\n \"endTime\": \"2026-03-16T14:37:40.220183063Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000a741e0>: failed to create HCP cluster cluster-zstream-4-20-8kc9g7: failed waiting for cluster="cluster-zstream-4-20-8kc9g7" in resourcegroup="rg-zstream-upgrade-4-20-dcjhgg" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f", "name": "724b22a8-140c-4634-bdee-1da54bd44a5f", "status": "Failed", "startTime": "2026-03-16T14:36:41.379585924Z", "endTime": "2026-03-16T14:37:40.220183063Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-8kc9g7: failed waiting for cluster=\"cluster-zstream-4-20-8kc9g7\" in resourcegroup=\"rg-zstream-upgrade-4-20-dcjhgg\" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f\",\n \"name\": \"724b22a8-140c-4634-bdee-1da54bd44a5f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:41.379585924Z\",\n \"endTime\": \"2026-03-16T14:37:40.220183063Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a74180>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-8kc9g7\" in resourcegroup=\"rg-zstream-upgrade-4-20-dcjhgg\" to finish creating: GET https://rp.j5249280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/724b22a8-140c-4634-bdee-1da54bd44a5f\",\n \"name\": \"724b22a8-140c-4634-bdee-1da54bd44a5f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:41.379585924Z\",\n \"endTime\": \"2026-03-16T14:37:40.220183063Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-137/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014821e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-7q2hlp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007ebcc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-7q2hlp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007ebca0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-7q2hlp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007ebc60>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-7q2hlp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014821e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-7q2hlp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007ebcc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-7q2hlp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007ebca0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-7q2hlp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007ebc60>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-7q2hlp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00102e2e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-r72ngj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d0d6a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-r72ngj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d0d680>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-r72ngj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d0d660>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-r72ngj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00102e2e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-r72ngj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d0d6a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-r72ngj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d0d680>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-r72ngj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d0d660>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-r72ngj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001760228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-x85bgb" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0014a3000>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-x85bgb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014a2fe0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-x85bgb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014a2fc0>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-x85bgb\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001760228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-x85bgb" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0014a3000>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-x85bgb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014a2fe0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-x85bgb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014a2fc0>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-x85bgb\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009f64c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5rf8lf" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000617060>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5rf8lf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000617020>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5rf8lf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000617000>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5rf8lf\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009f64c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5rf8lf" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000617060>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5rf8lf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000617020>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5rf8lf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000617000>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5rf8lf\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:53:11 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000518580>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000518580>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00105c0f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-ml49jg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013aa120>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-ml49jg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013aa100>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-ml49jg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013aa0e0>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-ml49jg\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00105c0f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-ml49jg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013aa120>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-ml49jg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013aa100>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-ml49jg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013aa0e0>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-ml49jg\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010b4408>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listx2wbnq" in resourcegroup="ea-list-9vgbb7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013ed380>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listx2wbnq\" in resourcegroup=\"ea-list-9vgbb7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013ed360>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listx2wbnq\" in resourcegroup=\"ea-list-9vgbb7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013ed340>{ msg: "failed waiting for hcpCluster=\"ea-listx2wbnq\" in resourcegroup=\"ea-list-9vgbb7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010b4408>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listx2wbnq" in resourcegroup="ea-list-9vgbb7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013ed380>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listx2wbnq\" in resourcegroup=\"ea-list-9vgbb7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013ed360>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listx2wbnq\" in resourcegroup=\"ea-list-9vgbb7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013ed340>{ msg: "failed waiting for hcpCluster=\"ea-listx2wbnq\" in resourcegroup=\"ea-list-9vgbb7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:53:11 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009603d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-k2wbnb" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009e54e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2wbnb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009e54c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2wbnb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009e54a0>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2wbnb\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009603d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-k2wbnb" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009e54e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2wbnb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009e54c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2wbnb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009e54a0>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2wbnb\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:52:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a82c625 · merge a82c625	n/a	e2e	failed	★#4347 (merged)	12	e2e (12) ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:52:23 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc00103a220>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-kx5x82-jnbcth" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-kx5x82-jnbcth\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00103a200>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-kx5x82-jnbcth\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list depl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc00103a220>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-kx5x82-jnbcth" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-kx5x82-jnbcth\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00103a200>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-kx5x82-jnbcth\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-kx5x82-jnbcth/operationStatuses/08584279346745938075\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list depl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:52:23 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc000747480>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-kp57zw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-kp57zw/operationStatuses/08584279346142577420 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-fkbb62/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-kp57zw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-kp57zw/operationStatuses/08584279346142577420\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-fkbb62/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments ca... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc000747480>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-kp57zw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-kp57zw/operationStatuses/08584279346142577420 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-fkbb62/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-kp57zw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-kp57zw/operationStatuses/08584279346142577420\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-fkbb62/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments ca... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:52:23 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:79]: Unexpected error: <fmt.wrapError \| 0xc000b295c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-mbz82w" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mbz82w/operationStatuses/08584279348304925301 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-vgcgcw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-mbz82w\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mbz82w/operationStatuses/08584279348304925301\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-vgcgcw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:79]: Unexpected error: <fmt.wrapError \| 0xc000b295c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-mbz82w" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mbz82w/operationStatuses/08584279348304925301 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-vgcgcw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-mbz82w\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mbz82w/operationStatuses/08584279348304925301\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-vgcgcw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:52:23 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000d588a0>: failed to create HCP cluster cluster-zstream-4-21-jc9kxj: failed waiting for cluster="cluster-zstream-4-21-jc9kxj" in resourcegroup="rg-zstream-upgrade-4-21-wvxm8g" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896", "name": "a56758ae-0e75-4522-b7af-560b514e2896", "status": "Failed", "startTime": "2026-03-16T14:43:45.183122663Z", "endTime": "2026-03-16T14:44:27.640803003Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-jc9kxj: failed waiting for cluster=\"cluster-zstream-4-21-jc9kxj\" in resourcegroup=\"rg-zstream-upgrade-4-21-wvxm8g\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896\",\n \"name\": \"a56758ae-0e75-4522-b7af-560b514e2896\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:43:45.183122663Z\",\n \"endTime\": \"2026-03-16T14:44:27.640803003Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000d58880>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-jc9kxj\" in resourcegroup=\"rg-zstream-upgrade-4-21-wvxm8g\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896\",\n \"name\": \"a56758ae-0e75-4522-b7af-560b514e2896\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:43:45.183122663Z\",\n \"endTime\": \"2026-03-16T14:44:27.640803003Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000d588a0>: failed to create HCP cluster cluster-zstream-4-21-jc9kxj: failed waiting for cluster="cluster-zstream-4-21-jc9kxj" in resourcegroup="rg-zstream-upgrade-4-21-wvxm8g" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896", "name": "a56758ae-0e75-4522-b7af-560b514e2896", "status": "Failed", "startTime": "2026-03-16T14:43:45.183122663Z", "endTime": "2026-03-16T14:44:27.640803003Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-jc9kxj: failed waiting for cluster=\"cluster-zstream-4-21-jc9kxj\" in resourcegroup=\"rg-zstream-upgrade-4-21-wvxm8g\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896\",\n \"name\": \"a56758ae-0e75-4522-b7af-560b514e2896\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:43:45.183122663Z\",\n \"endTime\": \"2026-03-16T14:44:27.640803003Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000d58880>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-jc9kxj\" in resourcegroup=\"rg-zstream-upgrade-4-21-wvxm8g\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a56758ae-0e75-4522-b7af-560b514e2896\",\n \"name\": \"a56758ae-0e75-4522-b7af-560b514e2896\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:43:45.183122663Z\",\n \"endTime\": \"2026-03-16T14:44:27.640803003Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-160/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:52:23 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:77]: Unexpected error: <fmt.wrapError \| 0xc00075e540>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-engineering-hcp-cluster-dskt5n" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-engineering-hcp-cluster-dskt5n/operationStatuses/08584279347938669527 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-m8fxcd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-5borhjiq7mwd4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-engineering-hcp-cluster-dskt5n\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-engineering-hcp-cluster-dskt5n/operationStatuses/08584279347938669527\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-m8fxcd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-5borhjiq7mwd4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/su... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:77]: Unexpected error: <fmt.wrapError \| 0xc00075e540>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-engineering-hcp-cluster-dskt5n" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-engineering-hcp-cluster-dskt5n/operationStatuses/08584279347938669527 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-m8fxcd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-5borhjiq7mwd4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-engineering-hcp-cluster-dskt5n\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-engineering-hcp-cluster-dskt5n/operationStatuses/08584279347938669527\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-m8fxcd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-5borhjiq7mwd4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/su... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline 11:52:23 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc00133e288>: context deadline exceeded failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline { errs: [ <context.deadlineExceededError>{}, <fmt.wrapError \| 0xc001613460>{ msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline", err: <fmt.wrapError \| 0xc001613440>{ msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline", err: <errors.errorString \| 0xc001584ce0>{ s: "rate: Wait(n=1) would exceed context deadline", }, }, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc00133e288>: context deadline exceeded failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline { errs: [ <context.deadlineExceededError>{}, <fmt.wrapError \| 0xc001613460>{ msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline", err: <fmt.wrapError \| 0xc001613440>{ msg: "client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline", err: <errors.errorString \| 0xc001584ce0>{ s: "rate: Wait(n=1) would exceed context deadline", }, }, }, ], } occurred ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift 11:52:23 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc000762460>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-vd8tln" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b", "name": "f7490142-2b7c-41f4-8adb-9053aa528f9b", "status": "Failed", "startTime": "2026-03-16T14:47:50.22465609Z", "endTime": "2026-03-16T14:49:01.889796339Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-vd8tln/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-vd8tln\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\",\n \"name\": \"f7490142-2b7c-41f4-8adb-9053aa528f9b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:47:50.22465609Z\",\n \"endTime\": \"2026-03-16T14:49:01.889796339Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-vd8tln/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0007622a0>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-vd8tln\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\n--------------------------------------------------------------------------------\nRESP... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc000762460>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-vd8tln" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b", "name": "f7490142-2b7c-41f4-8adb-9053aa528f9b", "status": "Failed", "startTime": "2026-03-16T14:47:50.22465609Z", "endTime": "2026-03-16T14:49:01.889796339Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-vd8tln/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-vd8tln\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\",\n \"name\": \"f7490142-2b7c-41f4-8adb-9053aa528f9b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:47:50.22465609Z\",\n \"endTime\": \"2026-03-16T14:49:01.889796339Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-vd8tln/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0007622a0>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-vd8tln\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7490142-2b7c-41f4-8adb-9053aa528f9b\n--------------------------------------------------------------------------------\nRESP... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:52:23 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dd86c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listg9z744" in resourcegroup="ea-list-fsnkgk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00080f760>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listg9z744\" in resourcegroup=\"ea-list-fsnkgk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00080f740>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listg9z744\" in resourcegroup=\"ea-list-fsnkgk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00080f6e0>{ msg: "failed waiting for hcpCluster=\"ea-listg9z744\" in resourcegroup=\"ea-list-fsnkgk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dd86c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listg9z744" in resourcegroup="ea-list-fsnkgk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00080f760>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listg9z744\" in resourcegroup=\"ea-list-fsnkgk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00080f740>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listg9z744\" in resourcegroup=\"ea-list-fsnkgk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00080f6e0>{ msg: "failed waiting for hcpCluster=\"ea-listg9z744\" in resourcegroup=\"ea-list-fsnkgk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:52:23 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error: <fmt.wrapError \| 0xc000804c20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-jjxpmq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/08584279346744937155 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-nhnjjj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-jjxpmq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/08584279346744937155\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-nhnjjj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000804c00>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-jjxpmq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/0... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error: <fmt.wrapError \| 0xc000804c20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-jjxpmq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/08584279346744937155 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-nhnjjj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-jjxpmq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/08584279346744937155\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-nhnjjj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000804c00>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-jjxpmq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-jjxpmq/operationStatuses/0... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:52:23 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000a80240>: failed to create HCP cluster cluster-zstream-4-19-m26nbt: failed waiting for cluster="cluster-zstream-4-19-m26nbt" in resourcegroup="rg-zstream-upgrade-4-19-px567h" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b", "name": "17fbc5fc-1a07-4aca-b897-a66cb03b930b", "status": "Failed", "startTime": "2026-03-16T14:46:56.283509646Z", "endTime": "2026-03-16T14:48:13.403575067Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-m26nbt: failed waiting for cluster=\"cluster-zstream-4-19-m26nbt\" in resourcegroup=\"rg-zstream-upgrade-4-19-px567h\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b\",\n \"name\": \"17fbc5fc-1a07-4aca-b897-a66cb03b930b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:46:56.283509646Z\",\n \"endTime\": \"2026-03-16T14:48:13.403575067Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a80220>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-m26nbt\" in resourcegroup=\"rg-zstream-upgrade-4-19-px567h\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b\",\n \"name\": \"17fbc5fc-1a07-4aca-b897-a66cb03b930b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:46:56.283509646Z\",\n \"endTime\": \"2026-03-16T14:48:13.403575067Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrot... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000a80240>: failed to create HCP cluster cluster-zstream-4-19-m26nbt: failed waiting for cluster="cluster-zstream-4-19-m26nbt" in resourcegroup="rg-zstream-upgrade-4-19-px567h" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b", "name": "17fbc5fc-1a07-4aca-b897-a66cb03b930b", "status": "Failed", "startTime": "2026-03-16T14:46:56.283509646Z", "endTime": "2026-03-16T14:48:13.403575067Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-m26nbt: failed waiting for cluster=\"cluster-zstream-4-19-m26nbt\" in resourcegroup=\"rg-zstream-upgrade-4-19-px567h\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b\",\n \"name\": \"17fbc5fc-1a07-4aca-b897-a66cb03b930b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:46:56.283509646Z\",\n \"endTime\": \"2026-03-16T14:48:13.403575067Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a80220>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-m26nbt\" in resourcegroup=\"rg-zstream-upgrade-4-19-px567h\" to finish creating: GET https://rp.j1571840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/17fbc5fc-1a07-4aca-b897-a66cb03b930b\",\n \"name\": \"17fbc5fc-1a07-4aca-b897-a66cb03b930b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:46:56.283509646Z\",\n \"endTime\": \"2026-03-16T14:48:13.403575067Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-60/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrot... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled 11:52:23 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010561e0>: failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-hbjgwj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0011132c0>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-hbjgwj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011132a0>{ msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-hbjgwj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010561e0>: failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-hbjgwj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0011132c0>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-hbjgwj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011132a0>{ msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-hbjgwj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:52:23 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001858228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-ppwm4q" in resourcegroup="admin-credential-lifecycle-test-4qglmk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b876a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-ppwm4q\" in resourcegroup=\"admin-credential-lifecycle-test-4qglmk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b87660>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-ppwm4q\" in resourcegroup=\"admin-credential-lifecycle-test-4qglmk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b87640>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-ppwm4q\" in resourcegroup=\"admin-credential-lifecycle-test-4qglmk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001858228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-ppwm4q" in resourcegroup="admin-credential-lifecycle-test-4qglmk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b876a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-ppwm4q\" in resourcegroup=\"admin-credential-lifecycle-test-4qglmk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b87660>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-ppwm4q\" in resourcegroup=\"admin-credential-lifecycle-test-4qglmk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b87640>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-ppwm4q\" in resourcegroup=\"admin-credential-lifecycle-test-4qglmk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:51:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head eac14df · merge 131806c	n/a	provision	failed	#4351 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid 11:51:36 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph Show raw failure time=2026-03-16T14:04:14.431Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-16T14:04:16.944Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra time=2026-03-16T14:04:18.414Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=1525bda5783b284829b029d62126cb30fcf582c6f2a60075236dde3ced9a8150 time=2026-03-16T14:05:48.773Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4808448/providers/Microsoft.Resources/deployments/1525bda5783b284829b029d62126cb30fcf582c6f2a60075236dde3ced9a8150/operationStatuses/08584279370280120648\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4808448/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4808448/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j4808448\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 64a663a4-5838-466b-8471-066c963f3378:3/16/2026 2:05:37 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
11:50:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0e39107	n/a	e2e	failed	★#4355 (closed)	9	e2e (9) missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:50:47 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.004s. Expected success, but got an error: <errors.errorString \| 0xc0009ce3c0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.004s. Expected success, but got an error: <errors.errorString \| 0xc0009ce3c0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:50:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016161b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-rpr688" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ba3680>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-rpr688\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ba3640>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-rpr688\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ba3620>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-rpr688\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016161b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-rpr688" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ba3680>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-rpr688\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ba3640>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-rpr688\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ba3620>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-rpr688\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:50:47 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000896c80>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-9btg4b-t58z2t" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-9btg4b-t58z2t/operationStatuses/08584279346703677235 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-mkw9kr/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-az4uc6ctjgcpo\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-249/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-9btg4b-t58z2t\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-9btg4b-t58z2t/operationStatuses/08584279346703677235\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-mkw9kr/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-az4uc6ctjgcpo\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-249/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignme... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000896c80>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-9btg4b-t58z2t" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-9btg4b-t58z2t/operationStatuses/08584279346703677235 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-mkw9kr/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-az4uc6ctjgcpo\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-249/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-9btg4b-t58z2t\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-9btg4b-t58z2t/operationStatuses/08584279346703677235\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-mkw9kr/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-az4uc6ctjgcpo\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-249/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignme... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:50:47 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0012d3758>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-cfx5gd" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000dee6e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-cfx5gd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000dee6c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-cfx5gd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000dee6a0>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-cfx5gd\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0012d3758>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-cfx5gd" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000dee6e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-cfx5gd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000dee6c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-cfx5gd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000dee6a0>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-cfx5gd\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:50:47 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010424b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-fvpr9k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007b1c80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-fvpr9k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007b1c60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-fvpr9k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007b1c40>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-fvpr9k\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010424b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-fvpr9k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007b1c80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-fvpr9k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007b1c60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-fvpr9k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007b1c40>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-fvpr9k\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:50:47 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e70258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-qxcp4w" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a11380>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-qxcp4w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a11360>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-qxcp4w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a11340>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-qxcp4w\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e70258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-qxcp4w" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a11380>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-qxcp4w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a11360>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-qxcp4w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a11340>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-qxcp4w\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:50:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009f8600>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-jff46k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00100b960>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-jff46k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00100b940>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-jff46k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00100b920>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-jff46k\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009f8600>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-jff46k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00100b960>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-jff46k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00100b940>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-jff46k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00100b920>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-jff46k\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:50:47 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000b184c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-tf444c-599hwd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-tf444c-599hwd/operationStatuses/08584279346098647612 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-196/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-gk2sxr/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-imkpyhgupivhu\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-21-tf444c-599hwd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-tf444c-599hwd/operationStatuses/08584279346098647612\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-196/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-gk2sxr/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-imkpyhgupivhu\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000b184c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-tf444c-599hwd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-tf444c-599hwd/operationStatuses/08584279346098647612 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-196/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-gk2sxr/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-imkpyhgupivhu\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-21-tf444c-599hwd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-tf444c-599hwd/operationStatuses/08584279346098647612\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-196/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-gk2sxr/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-imkpyhgupivhu\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:50:47 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000cd02a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-25sdtx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000761960>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-25sdtx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000761940>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-25sdtx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000761920>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-25sdtx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000cd02a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-25sdtx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000761960>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-25sdtx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000761940>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-25sdtx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000761920>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-25sdtx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:48:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2e6c304	n/a	e2e	failed	★#4364 (open)	20	e2e (20) ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:48:59 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:82]: Unexpected error: <fmt.wrapError \| 0xc0008873e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-admin-cred-lifecycle-g7fgt6-79nm6m" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-g7fgt6-79nm6m/operationStatuses/08584279347852100320 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-162/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-8prhlz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-jjxzjxtwa342w\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-admin-cred-lifecycle-g7fgt6-79nm6m\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-g7fgt6-79nm6m/operationStatuses/08584279347852100320\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-162/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-8prhlz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-jjxzjxtwa342w\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:82]: Unexpected error: <fmt.wrapError \| 0xc0008873e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-admin-cred-lifecycle-g7fgt6-79nm6m" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-g7fgt6-79nm6m/operationStatuses/08584279347852100320 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-162/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-8prhlz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-jjxzjxtwa342w\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-admin-cred-lifecycle-g7fgt6-79nm6m\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-g7fgt6-79nm6m/operationStatuses/08584279347852100320\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-162/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-8prhlz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-jjxzjxtwa342w\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:48:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: Unexpected error: <fmt.wrapError \| 0xc0008ab440>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-9fgcjk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-9fgcjk/operationStatuses/08584279346675224659 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-qfvgcd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-disabled-image-registry-hcp-cluster-9fgcjk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-9fgcjk/operationStatuses/08584279346675224659\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-qfvgcd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008ab420>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-disabled-image-registry-hcp-cluster-9fgcjk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployme... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: Unexpected error: <fmt.wrapError \| 0xc0008ab440>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-9fgcjk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-9fgcjk/operationStatuses/08584279346675224659 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-qfvgcd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-disabled-image-registry-hcp-cluster-9fgcjk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-9fgcjk/operationStatuses/08584279346675224659\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-qfvgcd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008ab420>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-disabled-image-registry-hcp-cluster-9fgcjk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployme... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:59 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00099c8a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-5kt9r5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000fb7380>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-5kt9r5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fb7360>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-5kt9r5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fb7340>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-5kt9r5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00099c8a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-5kt9r5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000fb7380>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-5kt9r5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fb7360>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-5kt9r5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fb7340>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-5kt9r5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:48:59 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000917b70>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000917b70>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010ae0d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qxpbrx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d76c60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-qxpbrx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d76c40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-qxpbrx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d76c20>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-qxpbrx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010ae0d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qxpbrx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d76c60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-qxpbrx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d76c40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-qxpbrx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d76c20>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-qxpbrx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:48:59 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000f11820>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-d4mcm7-gjd5hv" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-d4mcm7-gjd5hv/operationStatuses/08584279347271369558 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-195/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-21-d4mcm7-gjd5hv\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-d4mcm7-gjd5hv/operationStatuses/08584279347271369558\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-195/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f117e0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-21-d4mcm7-gjd5hv\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-d4mcm7-gjd5hv/operationStatuses/08584279347271369558\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-195/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000f11820>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-d4mcm7-gjd5hv" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-d4mcm7-gjd5hv/operationStatuses/08584279347271369558 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-195/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-21-d4mcm7-gjd5hv\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-d4mcm7-gjd5hv/operationStatuses/08584279347271369558\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-195/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f117e0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-21-d4mcm7-gjd5hv\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-d4mcm7-gjd5hv/operationStatuses/08584279347271369558\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-195/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000eab398>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-csfq6k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00105cf00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-csfq6k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00105cee0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-csfq6k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00105cec0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-csfq6k\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000eab398>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-csfq6k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00105cf00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-csfq6k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00105cee0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-csfq6k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00105cec0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-csfq6k\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001784240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vxfmp6" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008b35a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vxfmp6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008b3580>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vxfmp6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008b3560>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vxfmp6\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001784240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vxfmp6" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008b35a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vxfmp6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008b3580>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vxfmp6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008b3560>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vxfmp6\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:59 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00187a1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-q5hmwx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000729080>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-q5hmwx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000729040>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-q5hmwx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000729000>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-q5hmwx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00187a1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-q5hmwx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000729080>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-q5hmwx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000729040>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-q5hmwx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000729000>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-q5hmwx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:59 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014521c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-pwhwps" in resourcegroup="rg-zstream-upgrade-4-20-kf4b25" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a4f5c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-pwhwps\" in resourcegroup=\"rg-zstream-upgrade-4-20-kf4b25\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a4f5a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-pwhwps\" in resourcegroup=\"rg-zstream-upgrade-4-20-kf4b25\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a4f580>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-pwhwps\" in resourcegroup=\"rg-zstream-upgrade-4-20-kf4b25\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014521c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-pwhwps" in resourcegroup="rg-zstream-upgrade-4-20-kf4b25" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a4f5c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-pwhwps\" in resourcegroup=\"rg-zstream-upgrade-4-20-kf4b25\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a4f5a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-pwhwps\" in resourcegroup=\"rg-zstream-upgrade-4-20-kf4b25\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a4f580>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-pwhwps\" in resourcegroup=\"rg-zstream-upgrade-4-20-kf4b25\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:48:59 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc000f1b940>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-dc7gqr" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-dc7gqr/operationStatuses/08584279346155999693 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-162/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-4nkcvc/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-dc7gqr\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-dc7gqr/operationStatuses/08584279346155999693\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-162/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-4nkcvc/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc000f1b940>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-dc7gqr" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-dc7gqr/operationStatuses/08584279346155999693 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-162/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-4nkcvc/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-dc7gqr\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-dc7gqr/operationStatuses/08584279346155999693\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-162/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-4nkcvc/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:48:59 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc001522cc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-zctskf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-zctskf/operationStatuses/08584279348466930659 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-280/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8mrvmw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-zctskf\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-zctskf/operationStatuses/08584279348466930659\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-280/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8mrvmw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments c... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc001522cc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-zctskf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-zctskf/operationStatuses/08584279348466930659 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-280/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8mrvmw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-zctskf\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-zctskf/operationStatuses/08584279348466930659\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-280/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8mrvmw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments c... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:48:59 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc0009e4aa0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-jrlm9l-9f77ll" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-jrlm9l-9f77ll/operationStatuses/08584279346044663550 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cznbgf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-kue5o72yqqisy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-175/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-jrlm9l-9f77ll\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-jrlm9l-9f77ll/operationStatuses/08584279346044663550\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cznbgf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-kue5o72yqqisy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc0009e4aa0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-jrlm9l-9f77ll" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-jrlm9l-9f77ll/operationStatuses/08584279346044663550 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cznbgf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-kue5o72yqqisy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-175/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-jrlm9l-9f77ll\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-jrlm9l-9f77ll/operationStatuses/08584279346044663550\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cznbgf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-kue5o72yqqisy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001066318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-m96cdz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007e5280>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-m96cdz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007e5240>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-m96cdz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007e5220>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-m96cdz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001066318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-m96cdz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007e5280>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-m96cdz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007e5240>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-m96cdz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007e5220>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-m96cdz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network 11:48:59 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapError \| 0xc0004e55a0>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-tblwrw" to finish creating: GET https://rp.j0309632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905", "name": "8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905", "status": "Failed", "startTime": "2026-03-16T14:37:56.629756101Z", "endTime": "2026-03-16T14:38:48.032996565Z", "error": { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-tblwrw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-tblwrw\" to finish creating: GET https://rp.j0309632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\",\n \"name\": \"8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:37:56.629756101Z\",\n \"endTime\": \"2026-03-16T14:38:48.032996565Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-tblwrw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0004e5560>{ msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-tblwrw\" to finish creating: GET https://rp.j0309632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\",\n \"name\": \"8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:37:56.629756101Z\",\n \"endTime\": \"2026-03-16T14:38:48.032996565Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-tblwrw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again aft... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapError \| 0xc0004e55a0>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-tblwrw" to finish creating: GET https://rp.j0309632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905", "name": "8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905", "status": "Failed", "startTime": "2026-03-16T14:37:56.629756101Z", "endTime": "2026-03-16T14:38:48.032996565Z", "error": { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-tblwrw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-tblwrw\" to finish creating: GET https://rp.j0309632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\",\n \"name\": \"8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:37:56.629756101Z\",\n \"endTime\": \"2026-03-16T14:38:48.032996565Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-tblwrw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0004e5560>{ msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-tblwrw\" to finish creating: GET https://rp.j0309632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\",\n \"name\": \"8f9c4e00-f5e2-4c0e-bb9e-7c76759fe905\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:37:56.629756101Z\",\n \"endTime\": \"2026-03-16T14:38:48.032996565Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-tblwrw/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again aft... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:48:59 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:79]: Unexpected error: <fmt.wrapError \| 0xc000d92820>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-rfdqjk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-rfdqjk/operationStatuses/08584279348903194143 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-dcdfsp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-195/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-rfdqjk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-rfdqjk/operationStatuses/08584279348903194143\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-dcdfsp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:79]: Unexpected error: <fmt.wrapError \| 0xc000d92820>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-rfdqjk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-rfdqjk/operationStatuses/08584279348903194143 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-dcdfsp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-195/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-rfdqjk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-rfdqjk/operationStatuses/08584279348903194143\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-dcdfsp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:48:59 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:150]: Unexpected error: <fmt.wrapError \| 0xc000e21a80>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-zdjpzc" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-zdjpzc/operationStatuses/08584279347877156231 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-xxn4s8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-patch-tags-cluster-zdjpzc\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-zdjpzc/operationStatuses/08584279347877156231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-xxn4s8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e21a60>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-patch-tags-cluster-zdjpzc\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-zdjpzc/operationStatuses/08584279347877156231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation fa... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:150]: Unexpected error: <fmt.wrapError \| 0xc000e21a80>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-zdjpzc" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-zdjpzc/operationStatuses/08584279347877156231 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-xxn4s8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-patch-tags-cluster-zdjpzc\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-zdjpzc/operationStatuses/08584279347877156231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-xxn4s8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e21a60>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-patch-tags-cluster-zdjpzc\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-zdjpzc/operationStatuses/08584279347877156231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation fa... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f901c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-qxdqml" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00059d900>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qxdqml\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00059d8e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qxdqml\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00059d8c0>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qxdqml\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f901c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-qxdqml" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00059d900>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qxdqml\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00059d8e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qxdqml\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00059d8c0>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qxdqml\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:59 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014a8198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listf78f8t" in resourcegroup="ea-list-prmcsh" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000c238c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listf78f8t\" in resourcegroup=\"ea-list-prmcsh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c238a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listf78f8t\" in resourcegroup=\"ea-list-prmcsh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c23880>{ msg: "failed waiting for hcpCluster=\"ea-listf78f8t\" in resourcegroup=\"ea-list-prmcsh\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014a8198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listf78f8t" in resourcegroup="ea-list-prmcsh" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000c238c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listf78f8t\" in resourcegroup=\"ea-list-prmcsh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c238a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listf78f8t\" in resourcegroup=\"ea-list-prmcsh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c23880>{ msg: "failed waiting for hcpCluster=\"ea-listf78f8t\" in resourcegroup=\"ea-list-prmcsh\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:59 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0013f4330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-mqg6sm" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000eb1740>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-mqg6sm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000eb1720>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-mqg6sm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000eb1700>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-mqg6sm\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0013f4330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-mqg6sm" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000eb1740>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-mqg6sm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000eb1720>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-mqg6sm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000eb1700>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-mqg6sm\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:48:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ecf4a8c	n/a	e2e	failed	⚠#4369 (open)	12	e2e (12) Deserializaion Error: no output from command 11:48:11 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure 382910 pc=0x2b98a38 crypto/tls.(Conn).readFromUntil(0xc000806e08, {0x81cbfc0, 0xc000d38128}, 0xc0003829d0?) /usr/local/go/src/crypto/tls/conn.go:841 +0xde fp=0xc0003829a0 sp=0xc000382968 pc=0x2d6e89e crypto/tls.(Conn).readRecordOrCCS(0xc000806e08, 0x0) /usr/local/go/src/crypto/tls/conn.go:630 +0x3db fp=0xc000382c18 sp=0xc0003829a0 pc=0x2d6b91b crypto/tls.(Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:592 crypto/tls.(Conn).Read(0xc000806e08, {0xc00110f000, 0x1000, 0x31b05e0?}) /usr/local/go/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000382c80 sp=0xc000382c18 pc=0x2d72205 bufio.(Reader).Read(0xc00022cfc0, {0xc00061e2e4, 0x9, 0x31be72e?}) /usr/local/go/src/bufio/bufio.go:245 +0x197 fp=0xc000382cb8 sp=0xc000382c80 pc=0x2c341f7 io.ReadAtLeast({0x81caba0, 0xc00022cfc0}, {0xc00061e2e4, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:335 +0x8e fp=0xc000382d00 sp=0xc000382cb8 pc=0x2b28aee io.ReadFull(...) /usr/local/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0xc00061e2e4, 0x9, 0xc000000507?}, {0x81caba0?, 0xc00022cfc0?}) /go/pkg/mod/golang.org/x/net@v0.49.0/http2/frame.go:242 +0x65 fp=0xc000382d50 sp=0xc000382d00 pc=0x31a7d85 golang.org/x/net/http2.(Framer).ReadFrameHeader(0xc00061e2a0) /go/pkg/mod/golang.org/x/net@v0.49.0/http2/frame.go:505 +0x6b fp=0xc000382dd8 sp=0xc000382d50 pc=0x31a894b golang.org/x/net/http2.(Framer).ReadFrame(0xc00061e2a0) /go/pkg/mod/golang.org/x/net@v0.49.0/http2/frame.go:564 +0x18 fp=0xc000382e18 sp=0xc000382dd8 pc=0x31a90b8 golang.org/x/net/http2.(clientConnReadLoop).run(0xc000382fa8) /go/pkg/mod/golang.org/x/net@v0.49.0/http2/transport.go:2258 +0xca fp=0xc000382f60 sp=0xc000382e18 pc=0x31bb66a golang.org/x/net/http2.(ClientConn).readLoop(0xc001688540) /go/pkg/mod/golang.org/x/net@v0.49.0/http2/transport.go:2127 +0x52 fp=0xc000382fc8 sp=0xc000382f60 pc=0x31ba9f2 golang.org/x/net/http2.(Transport).newClientConn.gowrap1() /go/pkg/mod/golang.org/x/net@v0.49.0/http2/transport.go:880 +0x25 fp=0xc000382fe0 sp=0xc000382fc8 pc=0x31b4185 runtime.goexit({}) /usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000382fe8 sp=0xc000382fe0 pc=0x2ad6921 created by golang.org/x/net/http2.(Transport).newClientConn in goroutine 144 /go/pkg/mod/golang.org/x/net@v0.49.0/http2/transport.go:880 +0xda5 goroutine 1383 gp=0xc00161cfc0 m=nil [sync.WaitGroup.Wait, 3 minutes]: runtime.gopark(0xc001729af0?, 0x2a657d4?, 0x20?, 0xb9?, 0x7ffb3a4a5d06?) /usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000d9ba90 sp=0xc000d9ba70 pc=0x2ace28e runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:466 runtime.semacquire1(0xc0014c56d0, 0x0, 0x1, 0x0, 0x19) /usr/local/go/src/runtime/sema.go:192 +0x21d fp=0xc000d9baf8 sp=0xc000d9ba90 pc=0x2aaa3dd sync.runtime_SemacquireWaitGroup(0xc001004c00?, 0xc0?) /usr/local/go/src/runtime/sema.go:114 +0x2e fp=0xc000d9bb30 sp=0xc000d9baf8 pc=0x2acf9ae sync.(WaitGroup).Wait(0xc0014c56c8) /usr/local/go/src/sync/waitgroup.go:206 +0x85 fp=0xc000d9bb58 sp=0xc000d9bb30 pc=0x2af2e05 golang.org/x/sync/errgroup.(Group).Wait(0xc0014c56c0) /go/pkg/mod/golang.org/x/sync@v0.19.0/errgroup/errgroup.go:56 +0x1e fp=0xc000d9bb78 sp=0xc000d9bb58 pc=0x376d67e github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters({0x7ffb3818b460, 0xc0016fe180}, 0xc001004540, {0xc000a86dc0, 0x1c}, 0x34630b8a000) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:319 +0x8ef fp=0xc000d9be20 sp=0xc000d9bb78 pc=0x4eb296f github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).cleanupResourceGroup(0xc0008a4d80, {0x7ffb3818b460, 0xc0016fe180}, {0xc000a86dc0, 0x1c}, 0x34630b8a000) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:433 +0x294 fp=0xc000d9bf28 sp=0xc000d9be20 pc=0x4ec7c74 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7ffb3818b460, 0xc0016fe180}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:294 +0x14d fp=0xc000d9bfc0 sp=0xc000d9bf28 pc=0x4ec67ad github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups.gowrap1() /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:303 +0x28 fp=0xc000d9bfe0 sp=0xc000d9bfc0 pc=0x4ec6628 runtime.goexit({}) /usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d9bfe8 sp=0xc000d9bfe0 pc=0x2ad6921 created by github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups in goroutine 1382 /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:287 +0xc5 rax 0xca rbx 0x0 rcx 0x2ad8743 rdx 0x0 rdi 0xb361a90 rsi 0x80 rbp 0x7ffd9affcdd0 rsp 0x7ffd9affcd88 r8 0x0 r9 0x0 r10 0x0 r11 0x286 r12 0x8b r13 0x1 r14 0xb35f9a0 r15 0x1 rip 0x2ad8741 rflags 0x286 cs 0x33 fs 0x0 gs 0x0 Command Error: exit status 2 Deserializaion Error: no output from command 382910 pc=0x2b98a38 crypto/tls.(Conn).readFromUntil(0xc000806e08, {0x81cbfc0, 0xc000d38128}, 0xc0003829d0?) /usr/local/go/src/crypto/tls/conn.go:841 +0xde fp=0xc0003829a0 sp=0xc000382968 pc=0x2d6e89e crypto/tls.(Conn).readRecordOrCCS(0xc000806e08, 0x0) /usr/local/go/src/crypto/tls/conn.go:630 +0x3db fp=0xc000382c18 sp=0xc0003829a0 pc=0x2d6b91b crypto/tls.(Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:592 crypto/tls.(Conn).Read(0xc000806e08, {0xc00110f000, 0x1000, 0x31b05e0?}) /usr/local/go/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000382c80 sp=0xc000382c18 pc=0x2d72205 bufio.(Reader).Read(0xc00022cfc0, {0xc00061e2e4, 0x9, 0x31be72e?}) /usr/local/go/src/bufio/bufio.go:245 +0x197 fp=0xc000382cb8 sp=0xc000382c80 pc=0x2c341f7 io.ReadAtLeast({0x81caba0, 0xc00022cfc0}, {0xc00061e2e4, 0x9, 0x9}, 0x9) /usr/local/go/src/io/io.go:335 +0x8e fp=0xc000382d00 sp=0xc000382cb8 pc=0x2b28aee io.ReadFull(...) /usr/local/go/src/io/io.go:354 golang.org/x/net/http2.readFrameHeader({0xc00061e2e4, 0x9, 0xc000000507?}, {0x81caba0?, 0xc00022cfc0?}) /go/pkg/mod/golang.org/x/net@v0.49.0/http2/frame.go:242 +0x65 fp=0xc000382d50 sp=0xc000382d00 pc=0x31a7d85 golang.org/x/net/http2.(Framer).ReadFrameHeader(0xc00061e2a0) /go/pkg/mod/golang.org/x/net@v0.49.0/http2/frame.go:505 +0x6b fp=0xc000382dd8 sp=0xc000382d50 pc=0x31a894b golang.org/x/net/http2.(Framer).ReadFrame(0xc00061e2a0) /go/pkg/mod/golang.org/x/net@v0.49.0/http2/frame.go:564 +0x18 fp=0xc000382e18 sp=0xc000382dd8 pc=0x31a90b8 golang.org/x/net/http2.(clientConnReadLoop).run(0xc000382fa8) /go/pkg/mod/golang.org/x/net@v0.49.0/http2/transport.go:2258 +0xca fp=0xc000382f60 sp=0xc000382e18 pc=0x31bb66a golang.org/x/net/http2.(ClientConn).readLoop(0xc001688540) /go/pkg/mod/golang.org/x/net@v0.49.0/http2/transport.go:2127 +0x52 fp=0xc000382fc8 sp=0xc000382f60 pc=0x31ba9f2 golang.org/x/net/http2.(Transport).newClientConn.gowrap1() /go/pkg/mod/golang.org/x/net@v0.49.0/http2/transport.go:880 +0x25 fp=0xc000382fe0 sp=0xc000382fc8 pc=0x31b4185 runtime.goexit({}) /usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000382fe8 sp=0xc000382fe0 pc=0x2ad6921 created by golang.org/x/net/http2.(Transport).newClientConn in goroutine 144 /go/pkg/mod/golang.org/x/net@v0.49.0/http2/transport.go:880 +0xda5 goroutine 1383 gp=0xc00161cfc0 m=nil [sync.WaitGroup.Wait, 3 minutes]: runtime.gopark(0xc001729af0?, 0x2a657d4?, 0x20?, 0xb9?, 0x7ffb3a4a5d06?) /usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc000d9ba90 sp=0xc000d9ba70 pc=0x2ace28e runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:466 runtime.semacquire1(0xc0014c56d0, 0x0, 0x1, 0x0, 0x19) /usr/local/go/src/runtime/sema.go:192 +0x21d fp=0xc000d9baf8 sp=0xc000d9ba90 pc=0x2aaa3dd sync.runtime_SemacquireWaitGroup(0xc001004c00?, 0xc0?) /usr/local/go/src/runtime/sema.go:114 +0x2e fp=0xc000d9bb30 sp=0xc000d9baf8 pc=0x2acf9ae sync.(WaitGroup).Wait(0xc0014c56c8) /usr/local/go/src/sync/waitgroup.go:206 +0x85 fp=0xc000d9bb58 sp=0xc000d9bb30 pc=0x2af2e05 golang.org/x/sync/errgroup.(Group).Wait(0xc0014c56c0) /go/pkg/mod/golang.org/x/sync@v0.19.0/errgroup/errgroup.go:56 +0x1e fp=0xc000d9bb78 sp=0xc000d9bb58 pc=0x376d67e github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters({0x7ffb3818b460, 0xc0016fe180}, 0xc001004540, {0xc000a86dc0, 0x1c}, 0x34630b8a000) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:319 +0x8ef fp=0xc000d9be20 sp=0xc000d9bb78 pc=0x4eb296f github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).cleanupResourceGroup(0xc0008a4d80, {0x7ffb3818b460, 0xc0016fe180}, {0xc000a86dc0, 0x1c}, 0x34630b8a000) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:433 +0x294 fp=0xc000d9bf28 sp=0xc000d9be20 pc=0x4ec7c74 github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7ffb3818b460, 0xc0016fe180}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:294 +0x14d fp=0xc000d9bfc0 sp=0xc000d9bf28 pc=0x4ec67ad github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups.gowrap1() /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:303 +0x28 fp=0xc000d9bfe0 sp=0xc000d9bfc0 pc=0x4ec6628 runtime.goexit({}) /usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d9bfe8 sp=0xc000d9bfe0 pc=0x2ad6921 created by github.com/Azure/ARO-HCP/test/util/framework.(perItOrDescribeTestContext).CleanupResourceGroups in goroutine 1382 /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:287 +0xc5 rax 0xca rbx 0x0 rcx 0x2ad8743 rdx 0x0 rdi 0xb361a90 rsi 0x80 rbp 0x7ffd9affcdd0 rsp 0x7ffd9affcd88 r8 0x0 r9 0x0 r10 0x0 r11 0x286 r12 0x8b r13 0x1 r14 0xb35f9a0 r15 0x1 rip 0x2ad8741 rflags 0x286 cs 0x33 fs 0x0 gs 0x0 Command Error: exit status 2 Deserializaion Error: no output from command failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:11 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001784210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-mjbrxh" in resourcegroup="admin-credential-lifecycle-test-9znq98" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000fec140>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-mjbrxh\" in resourcegroup=\"admin-credential-lifecycle-test-9znq98\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fec120>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-mjbrxh\" in resourcegroup=\"admin-credential-lifecycle-test-9znq98\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fec100>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-mjbrxh\" in resourcegroup=\"admin-credential-lifecycle-test-9znq98\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001784210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-mjbrxh" in resourcegroup="admin-credential-lifecycle-test-9znq98" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000fec140>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-mjbrxh\" in resourcegroup=\"admin-credential-lifecycle-test-9znq98\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fec120>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-mjbrxh\" in resourcegroup=\"admin-credential-lifecycle-test-9znq98\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fec100>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-mjbrxh\" in resourcegroup=\"admin-credential-lifecycle-test-9znq98\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:11 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014ea120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-b99kmv" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f9ce00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-b99kmv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f9cdc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-b99kmv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f9cda0>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-b99kmv\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014ea120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-b99kmv" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f9ce00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-b99kmv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f9cdc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-b99kmv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f9cda0>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-b99kmv\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:11 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016d0348>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-gqdks9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00130a2c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-gqdks9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00130a2a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-gqdks9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00130a280>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-gqdks9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016d0348>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-gqdks9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00130a2c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-gqdks9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00130a2a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-gqdks9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00130a280>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-gqdks9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:11 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d9c270>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-2tztp4" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d8d4e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-2tztp4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d8d4c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-2tztp4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d8d4a0>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-2tztp4\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d9c270>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-2tztp4" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d8d4e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-2tztp4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d8d4c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-2tztp4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d8d4a0>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-2tztp4\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001170348>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vfl76k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0005904a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vfl76k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000590460>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vfl76k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000590440>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vfl76k\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001170348>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vfl76k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0005904a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vfl76k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000590460>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vfl76k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000590440>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vfl76k\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail <ip>: A node timeout occurred 11:48:11 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [:0]: A node timeout occurred fail [:0]: A node timeout occurred failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:11 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0007c6ab0>: failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-n5jtqq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0010cec40>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-n5jtqq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0010cec20>{ msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-n5jtqq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0007c6ab0>: failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-n5jtqq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0010cec40>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-n5jtqq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0010cec20>{ msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-n5jtqq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:11 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f32270>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-8rvdx2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00077b180>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-8rvdx2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00077b160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-8rvdx2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00077b140>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-8rvdx2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f32270>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-8rvdx2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00077b180>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-8rvdx2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00077b160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-8rvdx2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00077b140>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-8rvdx2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d7a300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-xfpb7p" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001075100>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-xfpb7p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0010750e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-xfpb7p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0010750c0>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-xfpb7p\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d7a300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-xfpb7p" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001075100>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-xfpb7p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0010750e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-xfpb7p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0010750c0>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-xfpb7p\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:48:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010ca258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-9zglhj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007938e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-9zglhj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007938c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-9zglhj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007938a0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-9zglhj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010ca258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-9zglhj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007938e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-9zglhj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007938c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-9zglhj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007938a0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-9zglhj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred route host was never found: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host 11:48:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error: <fmt.wrapError \| 0xc000e143c0>: route host was never found: Get "https://agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud": dial tcp: lookup agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud on 172.30.0.10:53: no such host { msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud\": dial tcp: lookup agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud on 172.30.0.10:53: no such host", err: <url.Error \| 0xc000d7f8c0>{ Op: "Get", URL: "https://agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud", Err: <net.OpError \| 0xc000d72550>{ Op: "dial", Net: "tcp", Source: nil, Addr: nil, Err: <net.DNSError \| 0xc000a534a0>{ UnwrapErr: nil, Err: "no such host", Name: "agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud", Server: "172.30.0.10:53", IsTimeout: false, IsTemporary: false, IsNotFound: true, }, }, }, } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error: <fmt.wrapError \| 0xc000e143c0>: route host was never found: Get "https://agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud": dial tcp: lookup agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud on 172.30.0.10:53: no such host { msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud\": dial tcp: lookup agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud on 172.30.0.10:53: no such host", err: <url.Error \| 0xc000d7f8c0>{ Op: "Get", URL: "https://agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud", Err: <net.OpError \| 0xc000d72550>{ Op: "dial", Net: "tcp", Source: nil, Addr: nil, Err: <net.DNSError \| 0xc000a534a0>{ UnwrapErr: nil, Err: "no such host", Name: "agnhost-e2e-serving-app-sckjc.apps.aro.u0e2e1n2t9u1a8h.4rck.j1302400.hcp.osadev.cloud", Server: "172.30.0.10:53", IsTimeout: false, IsTemporary: false, IsNotFound: true, }, }, }, } occurred
11:47:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2a2dae4	n/a	unknown	failed	#4370 (open)	1	non-artifact-backed failure (no junit artifacts)
11:46:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5f83abe	n/a	e2e	failed	★#4375 (closed)	17	e2e (17) ERROR CODE: InternalServerError; detail code 429; detail message Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds.; provider Microsoft.ManagedIdentity 11:46:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc0000f8ba0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-4ch82w" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d", "name": "26153cef-41b7-4eae-9a57-cc0a81a7c19d", "status": "Failed", "startTime": "2026-03-16T14:29:53.139498332Z", "endTime": "2026-03-16T14:31:10.125431656Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-4ch82w\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d\",\n \"name\": \"26153cef-41b7-4eae-9a57-cc0a81a7c19d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:53.139498332Z\",\n \"endTime\": \"2026-03-16T14:31:10.125431656Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0000f8b60>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-4ch82w\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d\",\n \"name\": \"26153cef-41b7-4eae-9a57-cc0a81a7c19d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:53.139498332Z\",\n \"endTime\": \"2026-03-16T14:31:10.125431656Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc0000f8ba0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-4ch82w" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d", "name": "26153cef-41b7-4eae-9a57-cc0a81a7c19d", "status": "Failed", "startTime": "2026-03-16T14:29:53.139498332Z", "endTime": "2026-03-16T14:31:10.125431656Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-4ch82w\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d\",\n \"name\": \"26153cef-41b7-4eae-9a57-cc0a81a7c19d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:53.139498332Z\",\n \"endTime\": \"2026-03-16T14:31:10.125431656Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0000f8b60>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-4ch82w\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/26153cef-41b7-4eae-9a57-cc0a81a7c19d\",\n \"name\": \"26153cef-41b7-4eae-9a57-cc0a81a7c19d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:53.139498332Z\",\n \"endTime\": \"2026-03-16T14:31:10.125431656Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fb4258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-7j9bkr" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000805a80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-7j9bkr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000805a60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-7j9bkr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000805a40>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-7j9bkr\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fb4258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-7j9bkr" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000805a80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-7j9bkr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000805a60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-7j9bkr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000805a40>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-7j9bkr\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed 11:46:23 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:152]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:152]: Cluster provisioning failed failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00100f0b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-h5jp46" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007937e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-h5jp46\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007937c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-h5jp46\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007937a0>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-h5jp46\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00100f0b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-h5jp46" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007937e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-h5jp46\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007937c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-h5jp46\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007937a0>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-h5jp46\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00111a3f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listnhkznp" in resourcegroup="ea-list-r74njm" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008a8600>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listnhkznp\" in resourcegroup=\"ea-list-r74njm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a85e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listnhkznp\" in resourcegroup=\"ea-list-r74njm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a85c0>{ msg: "failed waiting for hcpCluster=\"ea-listnhkznp\" in resourcegroup=\"ea-list-r74njm\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00111a3f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listnhkznp" in resourcegroup="ea-list-r74njm" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008a8600>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listnhkznp\" in resourcegroup=\"ea-list-r74njm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a85e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listnhkznp\" in resourcegroup=\"ea-list-r74njm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a85c0>{ msg: "failed waiting for hcpCluster=\"ea-listnhkznp\" in resourcegroup=\"ea-list-r74njm\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift 11:46:23 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc0007a4da0>: failed to create HCP cluster cluster-zstream-4-20-c4jlq6: failed waiting for cluster="cluster-zstream-4-20-c4jlq6" in resourcegroup="rg-zstream-upgrade-4-20-zx9bm6" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b34b3935-cea1-4830-80d1-79ff5a763ec5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b34b3935-cea1-4830-80d1-79ff5a763ec5", "name": "b34b3935-cea1-4830-80d1-79ff5a763ec5", "status": "Failed", "startTime": "2026-03-16T14:35:16.621213572Z", "endTime": "2026-03-16T14:36:17.621952691Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-129/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-129/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet-zstream-c4jlq6' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zx9bm6/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-c4jlq6\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-c4jlq6: failed waiting for cluster=\"cluster-zstream-4-20-c4jlq6\" in resourcegroup=\"rg-zstream-upgrade-4-20-zx9bm6\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b34b3935-cea1-4830-80d1-79ff5a763ec5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b34b3935-cea1-4830-80d1-79ff5a763ec5\",\n \"name\": \"b34b3935-cea1-4830-80d1-79ff5a763ec5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:35:16.621213572Z\",\n \"endTime\": \"2026-03-16T14:36:17.621952691Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-129/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-129/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-zstream-c4jlq6' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zx9bm6/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-c4jlq6\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0007a4d80>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-c4jlq6\" in resourcegroup=\"rg-zstream-upgrade-4-20-zx9bm6\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b34b3935-cea1-4830... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc0007a4da0>: failed to create HCP cluster cluster-zstream-4-20-c4jlq6: failed waiting for cluster="cluster-zstream-4-20-c4jlq6" in resourcegroup="rg-zstream-upgrade-4-20-zx9bm6" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b34b3935-cea1-4830-80d1-79ff5a763ec5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b34b3935-cea1-4830-80d1-79ff5a763ec5", "name": "b34b3935-cea1-4830-80d1-79ff5a763ec5", "status": "Failed", "startTime": "2026-03-16T14:35:16.621213572Z", "endTime": "2026-03-16T14:36:17.621952691Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-129/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-129/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet-zstream-c4jlq6' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zx9bm6/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-c4jlq6\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-c4jlq6: failed waiting for cluster=\"cluster-zstream-4-20-c4jlq6\" in resourcegroup=\"rg-zstream-upgrade-4-20-zx9bm6\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b34b3935-cea1-4830-80d1-79ff5a763ec5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b34b3935-cea1-4830-80d1-79ff5a763ec5\",\n \"name\": \"b34b3935-cea1-4830-80d1-79ff5a763ec5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:35:16.621213572Z\",\n \"endTime\": \"2026-03-16T14:36:17.621952691Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-129/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-129/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-zstream-c4jlq6' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-zx9bm6/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-c4jlq6\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0007a4d80>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-c4jlq6\" in resourcegroup=\"rg-zstream-upgrade-4-20-zx9bm6\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b34b3935-cea1-4830... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:46:23 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc00099d390>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc00099d390>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:46:23 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc000ba81e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-pxjp76" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pxjp76/operationStatuses/08584279348946056859 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-s8kh7j/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-pxjp76\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pxjp76/operationStatuses/08584279348946056859\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-s8kh7j/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc000ba81e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-pxjp76" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pxjp76/operationStatuses/08584279348946056859 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-s8kh7j/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-pxjp76\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pxjp76/operationStatuses/08584279348946056859\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-s8kh7j/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fa82e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-m5vdx7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013b66c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-m5vdx7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013b66a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-m5vdx7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001129260>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-m5vdx7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fa82e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-m5vdx7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013b66c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-m5vdx7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013b66a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-m5vdx7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001129260>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-m5vdx7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:46:23 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc0010d2140>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-92x7pp" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e", "name": "cb40865a-f041-407f-b523-d62555f0b51e", "status": "Failed", "startTime": "2026-03-16T14:35:15.551650129Z", "endTime": "2026-03-16T14:36:17.644274728Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-92x7pp\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e\",\n \"name\": \"cb40865a-f041-407f-b523-d62555f0b51e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:35:15.551650129Z\",\n \"endTime\": \"2026-03-16T14:36:17.644274728Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010d2120>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-92x7pp\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e\",\n \"name\": \"cb40865a-f041-407f-b523-d62555f0b51e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:35:15.551650129Z\",\n \"endTime\": \"2026-03-16T14:36:17.644274728Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc0010d2140>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-92x7pp" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e", "name": "cb40865a-f041-407f-b523-d62555f0b51e", "status": "Failed", "startTime": "2026-03-16T14:35:15.551650129Z", "endTime": "2026-03-16T14:36:17.644274728Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-92x7pp\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e\",\n \"name\": \"cb40865a-f041-407f-b523-d62555f0b51e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:35:15.551650129Z\",\n \"endTime\": \"2026-03-16T14:36:17.644274728Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010d2120>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-92x7pp\" to finish creating: GET https://rp.j0794112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb40865a-f041-407f-b523-d62555f0b51e\",\n \"name\": \"cb40865a-f041-407f-b523-d62555f0b51e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:35:15.551650129Z\",\n \"endTime\": \"2026-03-16T14:36:17.644274728Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d56540>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-nw2rtr" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00080d160>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-nw2rtr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00080d140>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-nw2rtr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00080d120>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-nw2rtr\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d56540>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-nw2rtr" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00080d160>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-nw2rtr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00080d140>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-nw2rtr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00080d120>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-nw2rtr\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00157a018>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-n29hzc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000021e20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n29hzc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000021e00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n29hzc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000021de0>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n29hzc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00157a018>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-n29hzc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000021e20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n29hzc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000021e00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n29hzc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000021de0>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-n29hzc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f301b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-q9z6gj" in resourcegroup="rg-zstream-upgrade-4-21-thqc42" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008074c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-q9z6gj\" in resourcegroup=\"rg-zstream-upgrade-4-21-thqc42\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008074a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-q9z6gj\" in resourcegroup=\"rg-zstream-upgrade-4-21-thqc42\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000807460>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-q9z6gj\" in resourcegroup=\"rg-zstream-upgrade-4-21-thqc42\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f301b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-q9z6gj" in resourcegroup="rg-zstream-upgrade-4-21-thqc42" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008074c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-q9z6gj\" in resourcegroup=\"rg-zstream-upgrade-4-21-thqc42\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008074a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-q9z6gj\" in resourcegroup=\"rg-zstream-upgrade-4-21-thqc42\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000807460>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-q9z6gj\" in resourcegroup=\"rg-zstream-upgrade-4-21-thqc42\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0011c81c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-9wjtwl" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d25040>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-9wjtwl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d25020>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-9wjtwl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d25000>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-9wjtwl\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0011c81c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-9wjtwl" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d25040>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-9wjtwl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d25020>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-9wjtwl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d25000>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-9wjtwl\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001194108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-6cnm8w" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000df6e60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-6cnm8w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000df6e20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-6cnm8w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000df6e00>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-6cnm8w\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001194108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-6cnm8w" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000df6e60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-6cnm8w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000df6e20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-6cnm8w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000df6e00>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-6cnm8w\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00112a180>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-x2g6p2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013770c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x2g6p2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013770a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x2g6p2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001377040>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x2g6p2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00112a180>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-x2g6p2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013770c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x2g6p2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013770a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x2g6p2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001377040>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x2g6p2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:46:23 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001026318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-7x952s" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000779c60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-7x952s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000779c20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-7x952s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000779c00>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-7x952s\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001026318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-7x952s" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000779c60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-7x952s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000779c20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-7x952s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000779c00>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-7x952s\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:45:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 07e421b · merge 07e421b	n/a	unknown	failed	#4385 (merged)	1	non-artifact-backed failure (no junit artifacts)
11:45:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head dacf876	n/a	unknown	failed	#4391 (open)	1	non-artifact-backed failure (no junit artifacts)
11:44:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head dcec1a0	n/a	e2e	failed	★#4393 (open)	11	e2e (11) ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network 11:44:23 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc001582140>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fwc64f" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea", "name": "647c9287-bdeb-4768-8590-42e0cce48fea", "status": "Failed", "startTime": "2026-03-16T12:39:38.774216229Z", "endTime": "2026-03-16T12:41:22.078116405Z", "error": { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fwc64f/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fwc64f\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea\",\n \"name\": \"647c9287-bdeb-4768-8590-42e0cce48fea\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:38.774216229Z\",\n \"endTime\": \"2026-03-16T12:41:22.078116405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fwc64f/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001582120>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fwc64f\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea\",\n \"name\": \"647c9287-bdeb-4768-8590-42e0cce48fea\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:38.774216229Z\",\n \"endTime\": \"2026-03-16T12:41:22.078116405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fwc64f/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc001582140>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fwc64f" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea", "name": "647c9287-bdeb-4768-8590-42e0cce48fea", "status": "Failed", "startTime": "2026-03-16T12:39:38.774216229Z", "endTime": "2026-03-16T12:41:22.078116405Z", "error": { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fwc64f/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fwc64f\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea\",\n \"name\": \"647c9287-bdeb-4768-8590-42e0cce48fea\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:38.774216229Z\",\n \"endTime\": \"2026-03-16T12:41:22.078116405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fwc64f/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001582120>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fwc64f\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/647c9287-bdeb-4768-8590-42e0cce48fea\",\n \"name\": \"647c9287-bdeb-4768-8590-42e0cce48fea\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:38.774216229Z\",\n \"endTime\": \"2026-03-16T12:41:22.078116405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fwc64f/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:44:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0017f2150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-4cvsn9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007addc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4cvsn9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007adda0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4cvsn9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007add80>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4cvsn9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0017f2150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-4cvsn9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007addc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4cvsn9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007adda0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4cvsn9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007add80>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4cvsn9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code 429; detail message Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds.; provider Microsoft.ManagedIdentity 11:44:23 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc000e4b8c0>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-vf4nkn" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf", "name": "652685d3-2aea-4847-ac21-8f15ad30cedf", "status": "Failed", "startTime": "2026-03-16T12:31:53.877846503Z", "endTime": "2026-03-16T12:32:42.631963218Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-vf4nkn\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf\",\n \"name\": \"652685d3-2aea-4847-ac21-8f15ad30cedf\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:31:53.877846503Z\",\n \"endTime\": \"2026-03-16T12:32:42.631963218Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e4b8a0>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-vf4nkn\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf\",\n \"name\": \"652685d3-2aea-4847-ac21-8f15ad30cedf\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:31:53.877846503Z\",\n \"endTime\": \"2026-03-16T12:32:42.631963218Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc000e4b8c0>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-vf4nkn" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf", "name": "652685d3-2aea-4847-ac21-8f15ad30cedf", "status": "Failed", "startTime": "2026-03-16T12:31:53.877846503Z", "endTime": "2026-03-16T12:32:42.631963218Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-vf4nkn\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf\",\n \"name\": \"652685d3-2aea-4847-ac21-8f15ad30cedf\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:31:53.877846503Z\",\n \"endTime\": \"2026-03-16T12:32:42.631963218Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 3 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e4b8a0>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-vf4nkn\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/652685d3-2aea-4847-ac21-8f15ad30cedf\",\n \"name\": \"652685d3-2aea-4847-ac21-8f15ad30cedf\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:31:53.877846503Z\",\n \"endTime\": \"2026-03-16T12:32:42.631963218Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred route host was never found: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host 11:44:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: Unexpected error: <fmt.wrapError \| 0xc000552bc0>: route host was never found: Get "https://agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud": dial tcp: lookup agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud on 172.30.0.10:53: no such host { msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud\": dial tcp: lookup agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud on 172.30.0.10:53: no such host", err: <url.Error \| 0xc000ea2d80>{ Op: "Get", URL: "https://agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud", Err: <net.OpError \| 0xc000d588c0>{ Op: "dial", Net: "tcp", Source: nil, Addr: nil, Err: <net.DNSError \| 0xc000ea51d0>{ UnwrapErr: nil, Err: "no such host", Name: "agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud", Server: "172.30.0.10:53", IsTimeout: false, IsTemporary: false, IsNotFound: true, }, }, }, } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: Unexpected error: <fmt.wrapError \| 0xc000552bc0>: route host was never found: Get "https://agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud": dial tcp: lookup agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud on 172.30.0.10:53: no such host { msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud\": dial tcp: lookup agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud on 172.30.0.10:53: no such host", err: <url.Error \| 0xc000ea2d80>{ Op: "Get", URL: "https://agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud", Err: <net.OpError \| 0xc000d588c0>{ Op: "dial", Net: "tcp", Source: nil, Addr: nil, Err: <net.DNSError \| 0xc000ea51d0>{ UnwrapErr: nil, Err: "no such host", Name: "agnhost-e2e-serving-app-9f7x5.apps.aro.l6q3l5t4y9r4i6k.15br.j8542976.hcp.osadev.cloud", Server: "172.30.0.10:53", IsTimeout: false, IsTemporary: false, IsNotFound: true, }, }, }, } occurred failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled 11:44:23 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000b2e0d8>: failed to cleanup resource group: failed waiting for resourcegroup="oidc-wi-rwm7g2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008a90e0>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"oidc-wi-rwm7g2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a90c0>{ msg: "failed waiting for resourcegroup=\"oidc-wi-rwm7g2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000b2e0d8>: failed to cleanup resource group: failed waiting for resourcegroup="oidc-wi-rwm7g2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008a90e0>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"oidc-wi-rwm7g2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a90c0>{ msg: "failed waiting for resourcegroup=\"oidc-wi-rwm7g2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:44:23 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dfe3a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-5f4chs" in resourcegroup="rg-zstream-upgrade-4-21-mpmgpd" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008a20e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-5f4chs\" in resourcegroup=\"rg-zstream-upgrade-4-21-mpmgpd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a20c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-5f4chs\" in resourcegroup=\"rg-zstream-upgrade-4-21-mpmgpd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a20a0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-5f4chs\" in resourcegroup=\"rg-zstream-upgrade-4-21-mpmgpd\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dfe3a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-5f4chs" in resourcegroup="rg-zstream-upgrade-4-21-mpmgpd" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008a20e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-5f4chs\" in resourcegroup=\"rg-zstream-upgrade-4-21-mpmgpd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a20c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-5f4chs\" in resourcegroup=\"rg-zstream-upgrade-4-21-mpmgpd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a20a0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-5f4chs\" in resourcegroup=\"rg-zstream-upgrade-4-21-mpmgpd\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:44:23 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e961f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-t6wj87" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001628360>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-t6wj87\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001628340>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-t6wj87\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001628320>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-t6wj87\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e961f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-t6wj87" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001628360>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-t6wj87\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001628340>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-t6wj87\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001628320>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-t6wj87\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:44:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0018900f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-qr7jf6" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009c6dc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qr7jf6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009c6da0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qr7jf6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009c6d80>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qr7jf6\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0018900f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-qr7jf6" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009c6dc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qr7jf6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009c6da0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qr7jf6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009c6d80>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-qr7jf6\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:44:23 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc000c88da0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bhdxzp" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43", "name": "67726f63-18de-4791-a065-5830cb428b43", "status": "Failed", "startTime": "2026-03-16T12:33:49.291317947Z", "endTime": "2026-03-16T12:35:21.638439699Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bhdxzp\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43\",\n \"name\": \"67726f63-18de-4791-a065-5830cb428b43\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:33:49.291317947Z\",\n \"endTime\": \"2026-03-16T12:35:21.638439699Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c88d80>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bhdxzp\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43\",\n \"name\": \"67726f63-18de-4791-a065-5830cb428b43\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:33:49.291317947Z\",\n \"endTime\": \"2026-03-16T12:35:21.638439699Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc000c88da0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-bhdxzp" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43", "name": "67726f63-18de-4791-a065-5830cb428b43", "status": "Failed", "startTime": "2026-03-16T12:33:49.291317947Z", "endTime": "2026-03-16T12:35:21.638439699Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bhdxzp\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43\",\n \"name\": \"67726f63-18de-4791-a065-5830cb428b43\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:33:49.291317947Z\",\n \"endTime\": \"2026-03-16T12:35:21.638439699Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c88d80>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-bhdxzp\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67726f63-18de-4791-a065-5830cb428b43\",\n \"name\": \"67726f63-18de-4791-a065-5830cb428b43\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:33:49.291317947Z\",\n \"endTime\": \"2026-03-16T12:35:21.638439699Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-128/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:44:23 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000c70ed0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-19-vvj692" in resourcegroup="rg-zstream-upgrade-4-19-sr4x5d" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001041560>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-vvj692\" in resourcegroup=\"rg-zstream-upgrade-4-19-sr4x5d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001041540>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-vvj692\" in resourcegroup=\"rg-zstream-upgrade-4-19-sr4x5d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001041520>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-vvj692\" in resourcegroup=\"rg-zstream-upgrade-4-19-sr4x5d\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000c70ed0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-19-vvj692" in resourcegroup="rg-zstream-upgrade-4-19-sr4x5d" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001041560>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-vvj692\" in resourcegroup=\"rg-zstream-upgrade-4-19-sr4x5d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001041540>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-vvj692\" in resourcegroup=\"rg-zstream-upgrade-4-19-sr4x5d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001041520>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-vvj692\" in resourcegroup=\"rg-zstream-upgrade-4-19-sr4x5d\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:44:23 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc000f1c5c0>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-x658kt" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5", "name": "c1bf58d2-c7a2-4659-9ac0-e943ec8118c5", "status": "Failed", "startTime": "2026-03-16T12:36:58.647009289Z", "endTime": "2026-03-16T12:38:12.224802026Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-x658kt\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\",\n \"name\": \"c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:58.647009289Z\",\n \"endTime\": \"2026-03-16T12:38:12.224802026Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f1c580>{ msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-x658kt\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\",\n \"name\": \"c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:58.647009289Z\",\n \"endTime\": \"2026-03-16T12:38:12.224802026Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrot... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc000f1c5c0>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-x658kt" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5", "name": "c1bf58d2-c7a2-4659-9ac0-e943ec8118c5", "status": "Failed", "startTime": "2026-03-16T12:36:58.647009289Z", "endTime": "2026-03-16T12:38:12.224802026Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-x658kt\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\",\n \"name\": \"c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:58.647009289Z\",\n \"endTime\": \"2026-03-16T12:38:12.224802026Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f1c580>{ msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-x658kt\" to finish creating: GET https://rp.j8542976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\",\n \"name\": \"c1bf58d2-c7a2-4659-9ac0-e943ec8118c5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:58.647009289Z\",\n \"endTime\": \"2026-03-16T12:38:12.224802026Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-104/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrot... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
11:43:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head a909a00 · merge 732c175	n/a	provision	failed	#4396 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code InvalidRequest; detail message The current utilization does not meet the criteria for both MaxTimeSeries and MaxEventsPerMinute quota requested. Please reach the required usage threshold of 50% of desired limit before requesting an increase, or request a limit increase of up to 200% of your current usage. 11:43:35 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph Show raw failure time=2026-03-16T12:04:32.942Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-03-16T12:04:35.189Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra time=2026-03-16T12:04:36.549Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=2ce6fd3928f382c8dddd47ceca6425de2f0fccdb442dfd6d7e3753e4871af3f7 time=2026-03-16T12:06:07.148Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j1123712/providers/Microsoft.Resources/deployments/2ce6fd3928f382c8dddd47ceca6425de2f0fccdb442dfd6d7e3753e4871af3f7/operationStatuses/08584279442097117108\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j1123712/providers/Microsoft.Resources/deployments/svc-monitor-ingestion-limits\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"InvalidRequest\\\",\\r\\n \\\"message\\\": \\\"The current utilization does not meet the criteria for both MaxTimeSeries and MaxEventsPerMinute quota requested. Please reach the required usage threshold of 50% of desired limit before requesting an increase, or request a limit increase of up to 200% of your current usage. For more details, see https://go.microsoft.com/fwlink/?linkid=2270124\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j1123712/providers/Microsoft.Resources/deployments/hcp-monitor-ingestion-limits\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"InvalidRequest\\\",\\r\\n \\\"message\\\": \\\"The current utilization does not meet the criteria for both MaxTimeSeries and MaxEventsPerMinute quota requested. Please reach the required usage threshold of 50% of desired limit before requesting an increase, or request a limit increase of up to 200% of your current usage. For more details, see https://go.microsoft.com/fwlink/?linkid=2270124\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
11:43:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 06408b4	n/a	e2e	failed	#4398 (closed)	13	e2e (13) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:43:11 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dac9d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-l9gr75" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e177c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-l9gr75\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e177a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-l9gr75\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e17780>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-l9gr75\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dac9d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-l9gr75" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e177c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-l9gr75\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e177a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-l9gr75\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e17780>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-l9gr75\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:43:11 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc001766b20>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-7dvc44" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f", "name": "437e5eb7-6380-44d3-a27a-311c39395c6f", "status": "Failed", "startTime": "2026-03-16T12:56:49.020928274Z", "endTime": "2026-03-16T12:57:52.704479743Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-7dvc44\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f\",\n \"name\": \"437e5eb7-6380-44d3-a27a-311c39395c6f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:49.020928274Z\",\n \"endTime\": \"2026-03-16T12:57:52.704479743Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001766b00>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-7dvc44\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f\",\n \"name\": \"437e5eb7-6380-44d3-a27a-311c39395c6f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:49.020928274Z\",\n \"endTime\": \"2026-03-16T12:57:52.704479743Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\":... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc001766b20>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-7dvc44" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f", "name": "437e5eb7-6380-44d3-a27a-311c39395c6f", "status": "Failed", "startTime": "2026-03-16T12:56:49.020928274Z", "endTime": "2026-03-16T12:57:52.704479743Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-7dvc44\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f\",\n \"name\": \"437e5eb7-6380-44d3-a27a-311c39395c6f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:49.020928274Z\",\n \"endTime\": \"2026-03-16T12:57:52.704479743Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001766b00>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-7dvc44\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/437e5eb7-6380-44d3-a27a-311c39395c6f\",\n \"name\": \"437e5eb7-6380-44d3-a27a-311c39395c6f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:49.020928274Z\",\n \"endTime\": \"2026-03-16T12:57:52.704479743Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-103/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\":... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:43:11 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc00091e220>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-mrc89s" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c", "name": "465467ce-7ab5-4614-a7c0-3f3868589f0c", "status": "Failed", "startTime": "2026-03-16T12:35:08.997993551Z", "endTime": "2026-03-16T12:36:36.827564904Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-mrc89s\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c\",\n \"name\": \"465467ce-7ab5-4614-a7c0-3f3868589f0c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:35:08.997993551Z\",\n \"endTime\": \"2026-03-16T12:36:36.827564904Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00091e200>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-mrc89s\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c\",\n \"name\": \"465467ce-7ab5-4614-a7c0-3f3868589f0c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:35:08.997993551Z\",\n \"endTime\": \"2026-03-16T12:36:36.827564904Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc00091e220>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-mrc89s" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c", "name": "465467ce-7ab5-4614-a7c0-3f3868589f0c", "status": "Failed", "startTime": "2026-03-16T12:35:08.997993551Z", "endTime": "2026-03-16T12:36:36.827564904Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-mrc89s\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c\",\n \"name\": \"465467ce-7ab5-4614-a7c0-3f3868589f0c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:35:08.997993551Z\",\n \"endTime\": \"2026-03-16T12:36:36.827564904Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00091e200>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-mrc89s\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/465467ce-7ab5-4614-a7c0-3f3868589f0c\",\n \"name\": \"465467ce-7ab5-4614-a7c0-3f3868589f0c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:35:08.997993551Z\",\n \"endTime\": \"2026-03-16T12:36:36.827564904Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-106/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:43:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00116e1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-jkvnzz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006fca00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-jkvnzz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006fc9c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-jkvnzz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006fc880>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-jkvnzz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00116e1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-jkvnzz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006fca00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-jkvnzz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006fc9c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-jkvnzz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006fc880>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-jkvnzz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:43:11 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000c7e3d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-xl2xk8" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0014d84e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-xl2xk8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014d84c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-xl2xk8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014d84a0>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-xl2xk8\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000c7e3d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-xl2xk8" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0014d84e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-xl2xk8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014d84c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-xl2xk8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014d84a0>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-xl2xk8\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:43:11 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000b85c20>: failed to create HCP cluster cluster-zstream-4-20-6r5wcr: failed waiting for cluster="cluster-zstream-4-20-6r5wcr" in resourcegroup="rg-zstream-upgrade-4-20-8cs2kg" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb", "name": "066bfaeb-683b-415d-afa1-18373f3e2bfb", "status": "Failed", "startTime": "2026-03-16T12:39:05.722653538Z", "endTime": "2026-03-16T12:40:27.237556224Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-6r5wcr: failed waiting for cluster=\"cluster-zstream-4-20-6r5wcr\" in resourcegroup=\"rg-zstream-upgrade-4-20-8cs2kg\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb\",\n \"name\": \"066bfaeb-683b-415d-afa1-18373f3e2bfb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:05.722653538Z\",\n \"endTime\": \"2026-03-16T12:40:27.237556224Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b85c00>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-6r5wcr\" in resourcegroup=\"rg-zstream-upgrade-4-20-8cs2kg\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb\",\n \"name\": \"066bfaeb-683b-415d-afa1-18373f3e2bfb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:05.722653538Z\",\n \"endTime\": \"2026-03-16T12:40:27.237556224Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000b85c20>: failed to create HCP cluster cluster-zstream-4-20-6r5wcr: failed waiting for cluster="cluster-zstream-4-20-6r5wcr" in resourcegroup="rg-zstream-upgrade-4-20-8cs2kg" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb", "name": "066bfaeb-683b-415d-afa1-18373f3e2bfb", "status": "Failed", "startTime": "2026-03-16T12:39:05.722653538Z", "endTime": "2026-03-16T12:40:27.237556224Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-6r5wcr: failed waiting for cluster=\"cluster-zstream-4-20-6r5wcr\" in resourcegroup=\"rg-zstream-upgrade-4-20-8cs2kg\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb\",\n \"name\": \"066bfaeb-683b-415d-afa1-18373f3e2bfb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:05.722653538Z\",\n \"endTime\": \"2026-03-16T12:40:27.237556224Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b85c00>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-6r5wcr\" in resourcegroup=\"rg-zstream-upgrade-4-20-8cs2kg\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/066bfaeb-683b-415d-afa1-18373f3e2bfb\",\n \"name\": \"066bfaeb-683b-415d-afa1-18373f3e2bfb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:05.722653538Z\",\n \"endTime\": \"2026-03-16T12:40:27.237556224Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:43:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e322b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-822n24" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d8d3c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-822n24\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d8d3a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-822n24\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d8d380>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-822n24\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e322b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-822n24" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d8d3c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-822n24\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d8d3a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-822n24\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d8d380>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-822n24\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:43:11 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc000f12680>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-tnp5xw" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6", "name": "e8b8c903-1c44-431d-90cc-722e05f4d5b6", "status": "Failed", "startTime": "2026-03-16T12:34:33.833275458Z", "endTime": "2026-03-16T12:35:37.744141088Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tnp5xw\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6\",\n \"name\": \"e8b8c903-1c44-431d-90cc-722e05f4d5b6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:33.833275458Z\",\n \"endTime\": \"2026-03-16T12:35:37.744141088Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f125e0>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tnp5xw\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6\",\n \"name\": \"e8b8c903-1c44-431d-90cc-722e05f4d5b6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:33.833275458Z\",\n \"endTime\": \"2026-03-16T12:35:37.744141088Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc000f12680>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-tnp5xw" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6", "name": "e8b8c903-1c44-431d-90cc-722e05f4d5b6", "status": "Failed", "startTime": "2026-03-16T12:34:33.833275458Z", "endTime": "2026-03-16T12:35:37.744141088Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tnp5xw\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6\",\n \"name\": \"e8b8c903-1c44-431d-90cc-722e05f4d5b6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:33.833275458Z\",\n \"endTime\": \"2026-03-16T12:35:37.744141088Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f125e0>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tnp5xw\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e8b8c903-1c44-431d-90cc-722e05f4d5b6\",\n \"name\": \"e8b8c903-1c44-431d-90cc-722e05f4d5b6\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:33.833275458Z\",\n \"endTime\": \"2026-03-16T12:35:37.744141088Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-124/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:43:11 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc000532660>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-22btj8" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83", "name": "86169469-3c78-4d45-a751-69b83d198d83", "status": "Failed", "startTime": "2026-03-16T12:56:13.574728771Z", "endTime": "2026-03-16T12:56:58.652053428Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-22btj8\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83\",\n \"name\": \"86169469-3c78-4d45-a751-69b83d198d83\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:13.574728771Z\",\n \"endTime\": \"2026-03-16T12:56:58.652053428Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000532640>{ msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-22btj8\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83\",\n \"name\": \"86169469-3c78-4d45-a751-69b83d198d83\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:13.574728771Z\",\n \"endTime\": \"2026-03-16T12:56:58.652053428Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc000532660>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-22btj8" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83", "name": "86169469-3c78-4d45-a751-69b83d198d83", "status": "Failed", "startTime": "2026-03-16T12:56:13.574728771Z", "endTime": "2026-03-16T12:56:58.652053428Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-22btj8\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83\",\n \"name\": \"86169469-3c78-4d45-a751-69b83d198d83\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:13.574728771Z\",\n \"endTime\": \"2026-03-16T12:56:58.652053428Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000532640>{ msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-22btj8\" to finish creating: GET https://rp.j1073920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86169469-3c78-4d45-a751-69b83d198d83\",\n \"name\": \"86169469-3c78-4d45-a751-69b83d198d83\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:13.574728771Z\",\n \"endTime\": \"2026-03-16T12:56:58.652053428Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-135/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:43:11 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016ce180>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-ldh25c" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b075c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ldh25c\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b07580>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ldh25c\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b07560>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ldh25c\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016ce180>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-ldh25c" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b075c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ldh25c\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b07580>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ldh25c\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b07560>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ldh25c\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:43:11 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e00120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-8khv2k" in resourcegroup="rg-zstream-upgrade-4-21-bxzggv" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000759a20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-8khv2k\" in resourcegroup=\"rg-zstream-upgrade-4-21-bxzggv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000759900>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-8khv2k\" in resourcegroup=\"rg-zstream-upgrade-4-21-bxzggv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007597c0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-8khv2k\" in resourcegroup=\"rg-zstream-upgrade-4-21-bxzggv\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e00120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-8khv2k" in resourcegroup="rg-zstream-upgrade-4-21-bxzggv" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000759a20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-8khv2k\" in resourcegroup=\"rg-zstream-upgrade-4-21-bxzggv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000759900>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-8khv2k\" in resourcegroup=\"rg-zstream-upgrade-4-21-bxzggv\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007597c0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-8khv2k\" in resourcegroup=\"rg-zstream-upgrade-4-21-bxzggv\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:43:11 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00108c318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-c7m2z6" in resourcegroup="admin-credential-lifecycle-test-2kddvf" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000805680>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-c7m2z6\" in resourcegroup=\"admin-credential-lifecycle-test-2kddvf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000805660>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-c7m2z6\" in resourcegroup=\"admin-credential-lifecycle-test-2kddvf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000805640>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-c7m2z6\" in resourcegroup=\"admin-credential-lifecycle-test-2kddvf\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00108c318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-c7m2z6" in resourcegroup="admin-credential-lifecycle-test-2kddvf" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000805680>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-c7m2z6\" in resourcegroup=\"admin-credential-lifecycle-test-2kddvf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000805660>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-c7m2z6\" in resourcegroup=\"admin-credential-lifecycle-test-2kddvf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000805640>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-c7m2z6\" in resourcegroup=\"admin-credential-lifecycle-test-2kddvf\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:43:11 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ed4330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vlv4fk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001341160>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vlv4fk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001341140>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vlv4fk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001341120>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vlv4fk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ed4330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vlv4fk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001341160>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vlv4fk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001341140>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vlv4fk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001341120>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-vlv4fk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:42:47	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head fca3d7b	n/a	e2e	failed	★#4402 (closed)	16	e2e (16) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f0e1c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-p7rzwx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0011a10c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-p7rzwx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011a10a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-p7rzwx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011a1080>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-p7rzwx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f0e1c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-p7rzwx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0011a10c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-p7rzwx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011a10a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-p7rzwx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011a1080>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-p7rzwx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000de57a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-v29lpc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00141cd40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-v29lpc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00141cd20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-v29lpc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00141cd00>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-v29lpc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000de57a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-v29lpc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00141cd40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-v29lpc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00141cd20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-v29lpc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00141cd00>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-v29lpc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e14150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-b9mmsd" in resourcegroup="rg-zstream-upgrade-4-21-hwl84h" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000797880>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-b9mmsd\" in resourcegroup=\"rg-zstream-upgrade-4-21-hwl84h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000797860>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-b9mmsd\" in resourcegroup=\"rg-zstream-upgrade-4-21-hwl84h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000797840>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-b9mmsd\" in resourcegroup=\"rg-zstream-upgrade-4-21-hwl84h\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e14150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-b9mmsd" in resourcegroup="rg-zstream-upgrade-4-21-hwl84h" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000797880>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-b9mmsd\" in resourcegroup=\"rg-zstream-upgrade-4-21-hwl84h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000797860>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-b9mmsd\" in resourcegroup=\"rg-zstream-upgrade-4-21-hwl84h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000797840>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-b9mmsd\" in resourcegroup=\"rg-zstream-upgrade-4-21-hwl84h\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ec02b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-lvrdg9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013eb2c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lvrdg9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013eb2a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lvrdg9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013eb280>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lvrdg9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ec02b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-lvrdg9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013eb2c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lvrdg9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013eb2a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lvrdg9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013eb280>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lvrdg9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00172e1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-7zq2cq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007818a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-7zq2cq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000781880>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-7zq2cq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000781860>{ msg: "failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-7zq2cq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00172e1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-7zq2cq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007818a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-7zq2cq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000781880>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-7zq2cq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000781860>{ msg: "failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-7zq2cq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014dc150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-hhv557" in resourcegroup="rg-zstream-upgrade-4-20-rrxqrj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e9cea0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-hhv557\" in resourcegroup=\"rg-zstream-upgrade-4-20-rrxqrj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e9ce80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-hhv557\" in resourcegroup=\"rg-zstream-upgrade-4-20-rrxqrj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e9ce60>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-hhv557\" in resourcegroup=\"rg-zstream-upgrade-4-20-rrxqrj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014dc150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-hhv557" in resourcegroup="rg-zstream-upgrade-4-20-rrxqrj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e9cea0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-hhv557\" in resourcegroup=\"rg-zstream-upgrade-4-20-rrxqrj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e9ce80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-hhv557\" in resourcegroup=\"rg-zstream-upgrade-4-20-rrxqrj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e9ce60>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-hhv557\" in resourcegroup=\"rg-zstream-upgrade-4-20-rrxqrj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00162a1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-qbt2j4" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000970200>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-qbt2j4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009701e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-qbt2j4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009701c0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-qbt2j4\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00162a1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-qbt2j4" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000970200>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-qbt2j4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009701e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-qbt2j4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009701c0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-qbt2j4\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:42:47 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error: <fmt.wrapError \| 0xc000e22140>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-7h22r5" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c", "name": "6b562372-8e68-442b-a67b-d54e8043f45c", "status": "Failed", "startTime": "2026-03-16T12:32:23.627486385Z", "endTime": "2026-03-16T12:33:16.905887614Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-7h22r5\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c\",\n \"name\": \"6b562372-8e68-442b-a67b-d54e8043f45c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:23.627486385Z\",\n \"endTime\": \"2026-03-16T12:33:16.905887614Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e22120>{ msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-7h22r5\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c\",\n \"name\": \"6b562372-8e68-442b-a67b-d54e8043f45c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:23.627486385Z\",\n \"endTime\": \"2026-03-16T12:33:16.905887614Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error: <fmt.wrapError \| 0xc000e22140>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-7h22r5" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c", "name": "6b562372-8e68-442b-a67b-d54e8043f45c", "status": "Failed", "startTime": "2026-03-16T12:32:23.627486385Z", "endTime": "2026-03-16T12:33:16.905887614Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-7h22r5\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c\",\n \"name\": \"6b562372-8e68-442b-a67b-d54e8043f45c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:23.627486385Z\",\n \"endTime\": \"2026-03-16T12:33:16.905887614Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e22120>{ msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-7h22r5\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b562372-8e68-442b-a67b-d54e8043f45c\",\n \"name\": \"6b562372-8e68-442b-a67b-d54e8043f45c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:23.627486385Z\",\n \"endTime\": \"2026-03-16T12:33:16.905887614Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-266/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0008bcd98>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-rrgcjp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a1fb00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rrgcjp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1fac0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rrgcjp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1faa0>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rrgcjp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0008bcd98>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-rrgcjp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a1fb00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rrgcjp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1fac0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rrgcjp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1faa0>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-rrgcjp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d90108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-n9zq8q" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007737e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-n9zq8q\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007737c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-n9zq8q\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007737a0>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-n9zq8q\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d90108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-n9zq8q" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007737e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-n9zq8q\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007737c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-n9zq8q\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007737a0>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-n9zq8q\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:42:47 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000772ec0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-nh4s5b" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549", "name": "6bc64492-e6a4-4fa8-bf06-17ab05a15549", "status": "Failed", "startTime": "2026-03-16T12:32:21.645989348Z", "endTime": "2026-03-16T12:33:16.903071324Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-nh4s5b\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549\",\n \"name\": \"6bc64492-e6a4-4fa8-bf06-17ab05a15549\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:21.645989348Z\",\n \"endTime\": \"2026-03-16T12:33:16.903071324Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000772e40>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-nh4s5b\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549\",\n \"name\": \"6bc64492-e6a4-4fa8-bf06-17ab05a15549\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:21.645989348Z\",\n \"endTime\": \"2026-03-16T12:33:16.903071324Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000772ec0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-nh4s5b" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549", "name": "6bc64492-e6a4-4fa8-bf06-17ab05a15549", "status": "Failed", "startTime": "2026-03-16T12:32:21.645989348Z", "endTime": "2026-03-16T12:33:16.903071324Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-nh4s5b\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549\",\n \"name\": \"6bc64492-e6a4-4fa8-bf06-17ab05a15549\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:21.645989348Z\",\n \"endTime\": \"2026-03-16T12:33:16.903071324Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000772e40>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-nh4s5b\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6bc64492-e6a4-4fa8-bf06-17ab05a15549\",\n \"name\": \"6bc64492-e6a4-4fa8-bf06-17ab05a15549\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:21.645989348Z\",\n \"endTime\": \"2026-03-16T12:33:16.903071324Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-245/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00082a360>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-l27lxr" in resourcegroup="admin-credential-lifecycle-test-kxjv9r" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00071de80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-l27lxr\" in resourcegroup=\"admin-credential-lifecycle-test-kxjv9r\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00071de20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-l27lxr\" in resourcegroup=\"admin-credential-lifecycle-test-kxjv9r\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00071dde0>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-l27lxr\" in resourcegroup=\"admin-credential-lifecycle-test-kxjv9r\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00082a360>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-l27lxr" in resourcegroup="admin-credential-lifecycle-test-kxjv9r" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00071de80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-l27lxr\" in resourcegroup=\"admin-credential-lifecycle-test-kxjv9r\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00071de20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-l27lxr\" in resourcegroup=\"admin-credential-lifecycle-test-kxjv9r\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00071dde0>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-l27lxr\" in resourcegroup=\"admin-credential-lifecycle-test-kxjv9r\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0004aa2d0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-h24hxm" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000813900>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-h24hxm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008138e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-h24hxm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008138c0>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-h24hxm\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0004aa2d0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-h24hxm" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000813900>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-h24hxm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008138e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-h24hxm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008138c0>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-h24hxm\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network 11:42:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapError \| 0xc0009646a0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tdsjqt" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d", "name": "7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d", "status": "Failed", "startTime": "2026-03-16T12:34:43.178140778Z", "endTime": "2026-03-16T12:36:17.674286842Z", "error": { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-tdsjqt/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-tdsjqt\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\",\n \"name\": \"7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:43.178140778Z\",\n \"endTime\": \"2026-03-16T12:36:17.674286842Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-tdsjqt/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000964680>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-tdsjqt\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\",\n \"name\": \"7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:43.178140778Z\",\n \"endTime\": \"2026-03-16T12:36:17.674286842Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-tdsjqt/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapError \| 0xc0009646a0>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tdsjqt" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d", "name": "7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d", "status": "Failed", "startTime": "2026-03-16T12:34:43.178140778Z", "endTime": "2026-03-16T12:36:17.674286842Z", "error": { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-tdsjqt/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-tdsjqt\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\",\n \"name\": \"7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:43.178140778Z\",\n \"endTime\": \"2026-03-16T12:36:17.674286842Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-tdsjqt/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000964680>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-tdsjqt\" to finish creating: GET https://rp.j0692480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\",\n \"name\": \"7c4fe4c6-eac0-470b-b2fb-9df4cab39e5d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:43.178140778Z\",\n \"endTime\": \"2026-03-16T12:36:17.674286842Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-tdsjqt/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010cc498>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-kkct5z" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a0d7a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-kkct5z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a0d780>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-kkct5z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a0d760>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-kkct5z\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010cc498>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-kkct5z" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a0d7a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-kkct5z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a0d780>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-kkct5z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a0d760>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-kkct5z\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:42:47 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00110a3a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-c7k8n8" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000fa9040>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-c7k8n8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fa9020>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-c7k8n8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fa9000>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-c7k8n8\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00110a3a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-c7k8n8" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000fa9040>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-c7k8n8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fa9020>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-c7k8n8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fa9000>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-c7k8n8\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:40:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head edbec11 · merge 80403bb	n/a	e2e	failed	★#4407 (merged)	9	e2e (9) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:40:59 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d12318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-jz9nh4" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f60d60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-jz9nh4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f60d40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-jz9nh4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f60d20>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-jz9nh4\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d12318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-jz9nh4" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f60d60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-jz9nh4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f60d40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-jz9nh4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f60d20>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-jz9nh4\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:40:59 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000b92258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-7lnqj8" in resourcegroup="rg-zstream-upgrade-4-21-z269g9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b2f7e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-7lnqj8\" in resourcegroup=\"rg-zstream-upgrade-4-21-z269g9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b2f7c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-7lnqj8\" in resourcegroup=\"rg-zstream-upgrade-4-21-z269g9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b2f7a0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-7lnqj8\" in resourcegroup=\"rg-zstream-upgrade-4-21-z269g9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000b92258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-7lnqj8" in resourcegroup="rg-zstream-upgrade-4-21-z269g9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b2f7e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-7lnqj8\" in resourcegroup=\"rg-zstream-upgrade-4-21-z269g9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b2f7c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-7lnqj8\" in resourcegroup=\"rg-zstream-upgrade-4-21-z269g9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b2f7a0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-7lnqj8\" in resourcegroup=\"rg-zstream-upgrade-4-21-z269g9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:40:59 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc0013642e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-q4vvzb" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-q4vvzb/operationStatuses/08584279343875914687 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vtr5fg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-214/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-q4vvzb\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-q4vvzb/operationStatuses/08584279343875914687\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vtr5fg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc0013642e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-q4vvzb" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-q4vvzb/operationStatuses/08584279343875914687 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vtr5fg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-214/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-q4vvzb\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-q4vvzb/operationStatuses/08584279343875914687\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-vtr5fg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:40:59 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000b54468>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-t7sblh" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a25a00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-t7sblh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a259a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-t7sblh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a25980>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-t7sblh\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000b54468>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-t7sblh" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a25a00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-t7sblh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a259a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-t7sblh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a25980>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-t7sblh\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:40:59 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000d14fa0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-8s2zrs-pvt4lx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-8s2zrs-pvt4lx/operationStatuses/08584279343979799486 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cm6xfn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ihxzyyx7uepx6\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-278/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-8s2zrs-pvt4lx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-8s2zrs-pvt4lx/operationStatuses/08584279343979799486\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cm6xfn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ihxzyyx7uepx6\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000d14fa0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-8s2zrs-pvt4lx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-8s2zrs-pvt4lx/operationStatuses/08584279343979799486 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cm6xfn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ihxzyyx7uepx6\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-278/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-8s2zrs-pvt4lx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-8s2zrs-pvt4lx/operationStatuses/08584279343979799486\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cm6xfn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ihxzyyx7uepx6\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:40:59 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001026570>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-52cdf9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00088bd40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-52cdf9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088bd00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-52cdf9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088bce0>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-52cdf9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001026570>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-52cdf9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00088bd40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-52cdf9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088bd00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-52cdf9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088bce0>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-52cdf9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:40:59 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc001621d00>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-jcvklv" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-jcvklv/operationStatuses/08584279343571105429 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-x4bdxj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-36/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-jcvklv\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-jcvklv/operationStatuses/08584279343571105429\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-x4bdxj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscri... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc001621d00>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-jcvklv" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-jcvklv/operationStatuses/08584279343571105429 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-x4bdxj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-36/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-jcvklv\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-jcvklv/operationStatuses/08584279343571105429\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-x4bdxj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscri... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:40:59 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000c58c20>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc000c58c20>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:40:59 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001560258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-46br52" in resourcegroup="rg-zstream-upgrade-4-20-bk6w4d" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000da12c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-46br52\" in resourcegroup=\"rg-zstream-upgrade-4-20-bk6w4d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000da12a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-46br52\" in resourcegroup=\"rg-zstream-upgrade-4-20-bk6w4d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000da1280>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-46br52\" in resourcegroup=\"rg-zstream-upgrade-4-20-bk6w4d\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001560258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-46br52" in resourcegroup="rg-zstream-upgrade-4-20-bk6w4d" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000da12c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-46br52\" in resourcegroup=\"rg-zstream-upgrade-4-20-bk6w4d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000da12a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-46br52\" in resourcegroup=\"rg-zstream-upgrade-4-20-bk6w4d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000da1280>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-46br52\" in resourcegroup=\"rg-zstream-upgrade-4-20-bk6w4d\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:40:11	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head cb8e965	n/a	provision	failed	#4411 (closed)	1	provision (1) ERROR: 'acr' is misspelled or not recognized by the system. 11:40:11 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Velero/global/mirror-azure-plugin-image · suite step graph Show raw failure time=2026-03-16T12:32:28.753Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image description="Step mirror-azure-plugin-image\n Kind: ImageMirror\n From {<nil> velero.azurePlugin.registry <nil>}:{<nil> velero.azurePlugin.repository <nil>}@{<nil> velero.azurePlugin.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-03-16T12:32:43.265Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Velero resourceGroup=global step=mirror-azure-plugin-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nFetch pull secret for source registry registry.redhat.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nFailed to log in to ACR arohcpsvcdev: ERROR: Error loading command module 'acs': deadlock detected by _ModuleLock('azure.core.pipeline.transport') at 140702637496304\nERROR: Error loading command module 'acr': cannot import name 'SpanKind' from partially initialized module 'azure.core.tracing' (most likely due to a circular import) (/usr/lib64/az/lib/python3.12/site-packages/azure/core/tracing/__init__.py)\nERROR: Error loading command module 'appservice': cannot import name 'PipelineClient' from 'azure.core' (/usr/lib64/az/lib/python3.12/site-packages/azure/core/__init__.py)\nERROR: 'acr' is misspelled or not recognized by the system.\n\nExamples from AI knowledge base:\nhttps://aka.ms/cli_ref\nRead more about the command in reference docs\n exit status 1"
11:39:45	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head af4e1e3	n/a	unknown	failed	#4412 (closed)	1	non-artifact-backed failure (no junit artifacts)
11:39:38	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head f259c28	n/a	e2e	failed	★#4413 (open)	4	e2e (4) missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:39:38 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc001035370>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.003s. Expected success, but got an error: <errors.errorString \| 0xc001035370>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:39:38 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error: <exported.ResponseError \| 0xc0013a6c90>: GET https://rp.j1451648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1a0dba70-b3ba-402b-b761-254d517d4678 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1a0dba70-b3ba-402b-b761-254d517d4678", "name": "1a0dba70-b3ba-402b-b761-254d517d4678", "status": "Failed", "startTime": "2026-03-16T13:01:52.936605636Z", "endTime": "2026-03-16T13:03:20.191663759Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-31/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-31/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "4f06b70e-ab59-4022-aa8d-b8b5d1af60b7", ], "Date": [ "Mon, 16 Mar 2026 13:03:23 GMT", ], "Content-Length": ["1559"], "X-Envoy-Upstream-Service-Time": ["11"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc00099a640>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 49, 97, 48, 100, 98, 97, 55, 48, 45, 98, 51, 98, 97, 45, 52, 48, 50, 98, 45, 98, 55, 54, 49, 45, 50, 53, 52, 100, 53, 49, 55, 100, 52, 54, 55, 56, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 49, 97, 48, 100, 98, 97, 55, 48, 45, 98, 51, 98, 97, 45, 52, 48, 50, 98, 45, 98, 55, 54, 49, 45, 50, 53, 52, 100, 53, 49, 55, 100, 52, 54, 55, 56, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 51, 58, 48, 49, 58, 53, 50, 46, 57, 51, 54, 54, 48, 53, 54, 51, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 51, 58, 48, 51, 58, 50, 48, 46, 49, 57, 49, 54, 54, 51, 55, 53, 57, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 51, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 11... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error: <exported.ResponseError \| 0xc0013a6c90>: GET https://rp.j1451648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1a0dba70-b3ba-402b-b761-254d517d4678 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1a0dba70-b3ba-402b-b761-254d517d4678", "name": "1a0dba70-b3ba-402b-b761-254d517d4678", "status": "Failed", "startTime": "2026-03-16T13:01:52.936605636Z", "endTime": "2026-03-16T13:03:20.191663759Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-31/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-31/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "4f06b70e-ab59-4022-aa8d-b8b5d1af60b7", ], "Date": [ "Mon, 16 Mar 2026 13:03:23 GMT", ], "Content-Length": ["1559"], "X-Envoy-Upstream-Service-Time": ["11"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc00099a640>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 49, 97, 48, 100, 98, 97, 55, 48, 45, 98, 51, 98, 97, 45, 52, 48, 50, 98, 45, 98, 55, 54, 49, 45, 50, 53, 52, 100, 53, 49, 55, 100, 52, 54, 55, 56, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 49, 97, 48, 100, 98, 97, 55, 48, 45, 98, 51, 98, 97, 45, 52, 48, 50, 98, 45, 98, 55, 54, 49, 45, 50, 53, 52, 100, 53, 49, 55, 100, 52, 54, 55, 56, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 51, 58, 48, 49, 58, 53, 50, 46, 57, 51, 54, 54, 48, 53, 54, 51, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 51, 58, 48, 51, 58, 50, 48, 46, 49, 57, 49, 54, 54, 51, 55, 53, 57, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 51, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 11... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:39:38 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc000492a60>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-6zfqcn" to finish creating: GET https://rp.j1451648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc", "name": "f143b52b-7509-495c-a365-0983468be9fc", "status": "Failed", "startTime": "2026-03-16T13:06:15.755983798Z", "endTime": "2026-03-16T13:07:20.629561299Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-6zfqcn\" to finish creating: GET https://rp.j1451648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc\",\n \"name\": \"f143b52b-7509-495c-a365-0983468be9fc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:06:15.755983798Z\",\n \"endTime\": \"2026-03-16T13:07:20.629561299Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000492a40>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-6zfqcn\" to finish creating: GET https://rp.j1451648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc\",\n \"name\": \"f143b52b-7509-495c-a365-0983468be9fc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:06:15.755983798Z\",\n \"endTime\": \"2026-03-16T13:07:20.629561299Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc000492a60>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-6zfqcn" to finish creating: GET https://rp.j1451648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc", "name": "f143b52b-7509-495c-a365-0983468be9fc", "status": "Failed", "startTime": "2026-03-16T13:06:15.755983798Z", "endTime": "2026-03-16T13:07:20.629561299Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-6zfqcn\" to finish creating: GET https://rp.j1451648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc\",\n \"name\": \"f143b52b-7509-495c-a365-0983468be9fc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:06:15.755983798Z\",\n \"endTime\": \"2026-03-16T13:07:20.629561299Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000492a40>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-6zfqcn\" to finish creating: GET https://rp.j1451648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f143b52b-7509-495c-a365-0983468be9fc\",\n \"name\": \"f143b52b-7509-495c-a365-0983468be9fc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:06:15.755983798Z\",\n \"endTime\": \"2026-03-16T13:07:20.629561299Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-53/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] 11:39:38 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.002s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Expected <[]string \| len:1, cap:1>: [ "image-registry (Available=False)", ] to be empty fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.002s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Expected <[]string \| len:1, cap:1>: [ "image-registry (Available=False)", ] to be empty
11:39:31	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2b8f05f	n/a	e2e	failed	★#4414 (closed)	16	e2e (16) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:31 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0015221e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listjbszpx" in resourcegroup="ea-list-ft7pjx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001139140>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listjbszpx\" in resourcegroup=\"ea-list-ft7pjx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001139120>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listjbszpx\" in resourcegroup=\"ea-list-ft7pjx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001139100>{ msg: "failed waiting for hcpCluster=\"ea-listjbszpx\" in resourcegroup=\"ea-list-ft7pjx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0015221e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listjbszpx" in resourcegroup="ea-list-ft7pjx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001139140>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listjbszpx\" in resourcegroup=\"ea-list-ft7pjx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001139120>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listjbszpx\" in resourcegroup=\"ea-list-ft7pjx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001139100>{ msg: "failed waiting for hcpCluster=\"ea-listjbszpx\" in resourcegroup=\"ea-list-ft7pjx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014f61c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-c97b6s" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ccd7a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-c97b6s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ccd780>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-c97b6s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ccd760>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-c97b6s\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014f61c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-c97b6s" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ccd7a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-c97b6s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ccd780>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-c97b6s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ccd760>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-c97b6s\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000190fa8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-hkbztn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000981d60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-hkbztn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000981d40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-hkbztn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000981d20>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-hkbztn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000190fa8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-hkbztn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000981d60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-hkbztn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000981d40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-hkbztn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000981d20>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-hkbztn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred client rate limiter Wait returned an error: context deadline exceeded 11:39:31 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc0010301b0>: context deadline exceeded failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded { errs: [ <context.deadlineExceededError>{}, <fmt.wrapError \| 0xc000797340>{ msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded", err: <fmt.wrapError \| 0xc000797300>{ msg: "client rate limiter Wait returned an error: context deadline exceeded", err: <context.deadlineExceededError>{}, }, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc0010301b0>: context deadline exceeded failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded { errs: [ <context.deadlineExceededError>{}, <fmt.wrapError \| 0xc000797340>{ msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded", err: <fmt.wrapError \| 0xc000797300>{ msg: "client rate limiter Wait returned an error: context deadline exceeded", err: <context.deadlineExceededError>{}, }, }, ], } occurred missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:39:31 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s. Expected success, but got an error: <errors.errorString \| 0xc000fa96a0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s. Expected success, but got an error: <errors.errorString \| 0xc000fa96a0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:31 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000a24780>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-w6lldn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000979be0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-w6lldn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000979bc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-w6lldn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000979b60>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-w6lldn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000a24780>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-w6lldn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000979be0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-w6lldn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000979bc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-w6lldn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000979b60>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-w6lldn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:39:31 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc000413360>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sdxgfv" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828", "name": "6883ff96-373a-49d2-8850-c1af1afe4828", "status": "Failed", "startTime": "2026-03-16T14:36:35.110062481Z", "endTime": "2026-03-16T14:37:50.877973227Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sdxgfv\" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828\",\n \"name\": \"6883ff96-373a-49d2-8850-c1af1afe4828\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:35.110062481Z\",\n \"endTime\": \"2026-03-16T14:37:50.877973227Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000413320>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sdxgfv\" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828\",\n \"name\": \"6883ff96-373a-49d2-8850-c1af1afe4828\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:35.110062481Z\",\n \"endTime\": \"2026-03-16T14:37:50.877973227Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc000413360>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sdxgfv" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828", "name": "6883ff96-373a-49d2-8850-c1af1afe4828", "status": "Failed", "startTime": "2026-03-16T14:36:35.110062481Z", "endTime": "2026-03-16T14:37:50.877973227Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sdxgfv\" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828\",\n \"name\": \"6883ff96-373a-49d2-8850-c1af1afe4828\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:35.110062481Z\",\n \"endTime\": \"2026-03-16T14:37:50.877973227Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000413320>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sdxgfv\" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6883ff96-373a-49d2-8850-c1af1afe4828\",\n \"name\": \"6883ff96-373a-49d2-8850-c1af1afe4828\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:35.110062481Z\",\n \"endTime\": \"2026-03-16T14:37:50.877973227Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:39:31 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc00150c520>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-pqknpk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pqknpk/operationStatuses/08584279348255055112 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-v6hvbv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-pqknpk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pqknpk/operationStatuses/08584279348255055112\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-v6hvbv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc00150c520>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-pqknpk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pqknpk/operationStatuses/08584279348255055112 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-v6hvbv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-pqknpk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-pqknpk/operationStatuses/08584279348255055112\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-v6hvbv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000a40498>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-fl5mzp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013d7200>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-fl5mzp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013d71e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-fl5mzp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013d71c0>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-fl5mzp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000a40498>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-fl5mzp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0013d7200>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-fl5mzp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013d71e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-fl5mzp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013d71c0>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-fl5mzp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:39:31 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc00049fa40>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-n6hv7n-4x29z4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-n6hv7n-4x29z4/operationStatuses/08584279346342706585 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-n6hv7n-4x29z4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-n6hv7n-4x29z4/operationStatuses/08584279346342706585\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00049fa20>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-n6hv7n-4x29z4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-n6hv7n-4x29z4/operationStatuses/08584279346342706585\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list depl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc00049fa40>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-n6hv7n-4x29z4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-n6hv7n-4x29z4/operationStatuses/08584279346342706585 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-n6hv7n-4x29z4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-n6hv7n-4x29z4/operationStatuses/08584279346342706585\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00049fa20>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-n6hv7n-4x29z4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-n6hv7n-4x29z4/operationStatuses/08584279346342706585\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-75/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list depl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:31 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014c0360>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-79p6sp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e4a6a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-79p6sp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e4a680>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-79p6sp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e4a660>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-79p6sp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014c0360>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-79p6sp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e4a6a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-79p6sp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e4a680>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-79p6sp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e4a660>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-79p6sp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network 11:39:31 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc0016254e0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-h89ld2" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e", "name": "ebe17b9b-b660-46f1-be75-e057609d910e", "status": "Failed", "startTime": "2026-03-16T14:47:24.104802433Z", "endTime": "2026-03-16T14:48:51.18681009Z", "error": { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-h89ld2/providers/Microsoft.Network/virtualNetworks/customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-h89ld2\" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e\",\n \"name\": \"ebe17b9b-b660-46f1-be75-e057609d910e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:47:24.104802433Z\",\n \"endTime\": \"2026-03-16T14:48:51.18681009Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-h89ld2/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0016254c0>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-h89ld2\" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e\",\n \"name\": \"ebe17b9b-b660-46f1-be75-e057609d910e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:47:24.104802433Z\",\n \"endTime\": \"2026-03-16T14:48:51.18681009Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-h89ld2/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc0016254e0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-h89ld2" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e", "name": "ebe17b9b-b660-46f1-be75-e057609d910e", "status": "Failed", "startTime": "2026-03-16T14:47:24.104802433Z", "endTime": "2026-03-16T14:48:51.18681009Z", "error": { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-h89ld2/providers/Microsoft.Network/virtualNetworks/customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-h89ld2\" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e\",\n \"name\": \"ebe17b9b-b660-46f1-be75-e057609d910e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:47:24.104802433Z\",\n \"endTime\": \"2026-03-16T14:48:51.18681009Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-h89ld2/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0016254c0>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-h89ld2\" to finish creating: GET https://rp.j3148160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ebe17b9b-b660-46f1-be75-e057609d910e\",\n \"name\": \"ebe17b9b-b660-46f1-be75-e057609d910e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:47:24.104802433Z\",\n \"endTime\": \"2026-03-16T14:48:51.18681009Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-h89ld2/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:31 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001804360>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-9b65vd" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a1c580>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-9b65vd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1c520>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-9b65vd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1c480>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-9b65vd\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001804360>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-9b65vd" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a1c580>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-9b65vd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1c520>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-9b65vd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1c480>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-9b65vd\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ff01e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-x7q5bp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f10de0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x7q5bp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f10dc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x7q5bp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f10da0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x7q5bp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ff01e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-x7q5bp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f10de0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x7q5bp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f10dc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x7q5bp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f10da0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-x7q5bp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:31 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ee6228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-5qv9g7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0014a6f40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5qv9g7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014a6f20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5qv9g7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014a6f00>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5qv9g7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ee6228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-5qv9g7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0014a6f40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5qv9g7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014a6f20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5qv9g7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014a6f00>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5qv9g7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:31 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e84198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-kqqw8g" in resourcegroup="rg-zstream-upgrade-4-21-t9b2n6" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001271080>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-kqqw8g\" in resourcegroup=\"rg-zstream-upgrade-4-21-t9b2n6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001271060>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-kqqw8g\" in resourcegroup=\"rg-zstream-upgrade-4-21-t9b2n6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001271040>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-kqqw8g\" in resourcegroup=\"rg-zstream-upgrade-4-21-t9b2n6\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e84198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-kqqw8g" in resourcegroup="rg-zstream-upgrade-4-21-t9b2n6" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001271080>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-kqqw8g\" in resourcegroup=\"rg-zstream-upgrade-4-21-t9b2n6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001271060>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-kqqw8g\" in resourcegroup=\"rg-zstream-upgrade-4-21-t9b2n6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001271040>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-kqqw8g\" in resourcegroup=\"rg-zstream-upgrade-4-21-t9b2n6\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0990c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:39:24	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 2d42e98 · merge 855eea8	n/a	unknown	failed	#4425 (merged)	1	non-artifact-backed failure (no junit artifacts)
11:39:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 599dd2c · merge 599dd2c	n/a	e2e	failed	★#4433 (merged)	16	e2e (16) All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] 11:39:08 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.001s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Expected <[]string \| len:1, cap:1>: [ "image-registry (Available=False)", ] to be empty fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.001s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Expected <[]string \| len:1, cap:1>: [ "image-registry (Available=False)", ] to be empty failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:08 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0017b2240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-hblwz4" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0017893e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-hblwz4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0017893c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-hblwz4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001789380>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-hblwz4\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0017b2240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-hblwz4" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0017893e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-hblwz4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0017893c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-hblwz4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001789380>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-hblwz4\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift 11:39:08 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc0008d6580>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-f6npnk" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4", "name": "5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4", "status": "Failed", "startTime": "2026-03-16T14:51:40.269113062Z", "endTime": "2026-03-16T14:53:13.013429678Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-f6npnk/providers/Microsoft.Network/virtualNetworks/customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-f6npnk\" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4\",\n \"name\": \"5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:51:40.269113062Z\",\n \"endTime\": \"2026-03-16T14:53:13.013429678Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-f6npnk/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008d6540>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-f6npnk\" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4\n---------------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc0008d6580>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-f6npnk" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4", "name": "5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4", "status": "Failed", "startTime": "2026-03-16T14:51:40.269113062Z", "endTime": "2026-03-16T14:53:13.013429678Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-f6npnk/providers/Microsoft.Network/virtualNetworks/customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-f6npnk\" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4\",\n \"name\": \"5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:51:40.269113062Z\",\n \"endTime\": \"2026-03-16T14:53:13.013429678Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-f6npnk/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-37/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008d6540>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-f6npnk\" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e1fe82d-4e07-4f03-a07b-d87cb2e29dd4\n---------------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001640168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-nz84xw" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00135ee40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-nz84xw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00135ee20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-nz84xw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00135ee00>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-nz84xw\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001640168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-nz84xw" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00135ee40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-nz84xw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00135ee20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-nz84xw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00135ee00>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-nz84xw\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:39:08 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc0011baba0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-9lh2mn-nbskpm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-9lh2mn-nbskpm/operationStatuses/08584279343902214935 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-x6l9xl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rj3ukhr2ls7pg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-37/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-9lh2mn-nbskpm\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-9lh2mn-nbskpm/operationStatuses/08584279343902214935\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-x6l9xl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rj3ukhr2ls7pg\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc0011baba0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-9lh2mn-nbskpm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-9lh2mn-nbskpm/operationStatuses/08584279343902214935 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-x6l9xl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rj3ukhr2ls7pg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-37/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-9lh2mn-nbskpm\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-9lh2mn-nbskpm/operationStatuses/08584279343902214935\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-x6l9xl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rj3ukhr2ls7pg\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:08 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f58330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-gccfmc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009319a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-gccfmc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000931980>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-gccfmc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000931960>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-gccfmc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f58330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-gccfmc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009319a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-gccfmc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000931980>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-gccfmc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000931960>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-gccfmc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:39:08 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:151]: Unexpected error: <fmt.wrapError \| 0xc0013f01e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster3-gck6vl" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-gck6vl/operationStatuses/08584279347703054161 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-qkqcp2/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-gck6vl\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-gck6vl/operationStatuses/08584279347703054161\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-qkqcp2/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0013f01c0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-gck6vl\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-gck6vl/operationStatuses/085842793477... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:151]: Unexpected error: <fmt.wrapError \| 0xc0013f01e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster3-gck6vl" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-gck6vl/operationStatuses/08584279347703054161 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-qkqcp2/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-gck6vl\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-gck6vl/operationStatuses/08584279347703054161\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-qkqcp2/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0013f01c0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-gck6vl\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-gck6vl/operationStatuses/085842793477... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001624198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-z64tpk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00097f180>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-z64tpk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00097f160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-z64tpk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00097f140>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-z64tpk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001624198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-z64tpk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00097f180>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-z64tpk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00097f160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-z64tpk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00097f140>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-z64tpk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:08 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000868480>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-hjjnp8" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e44fe0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-hjjnp8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e44fc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-hjjnp8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e44fa0>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-hjjnp8\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000868480>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-hjjnp8" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e44fe0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-hjjnp8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e44fc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-hjjnp8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e44fa0>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-hjjnp8\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:08 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e2e240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-8rxw7g" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001341760>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-8rxw7g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001341740>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-8rxw7g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001341720>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-8rxw7g\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e2e240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-8rxw7g" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001341760>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-8rxw7g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001341740>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-8rxw7g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001341720>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-8rxw7g\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000a345e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-22j64g" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e0d260>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-22j64g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e0d240>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-22j64g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e0d220>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-22j64g\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000a345e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-22j64g" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e0d260>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-22j64g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e0d240>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-22j64g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e0d220>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-22j64g\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:39:08 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc001621880>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-rwkvnl" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5", "name": "b71f3c20-bebb-45a9-8de6-fc37671e60f5", "status": "Failed", "startTime": "2026-03-16T14:55:54.425365996Z", "endTime": "2026-03-16T14:56:39.660041286Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-rwkvnl\" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5\",\n \"name\": \"b71f3c20-bebb-45a9-8de6-fc37671e60f5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:55:54.425365996Z\",\n \"endTime\": \"2026-03-16T14:56:39.660041286Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001621860>{ msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-rwkvnl\" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5\",\n \"name\": \"b71f3c20-bebb-45a9-8de6-fc37671e60f5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:55:54.425365996Z\",\n \"endTime\": \"2026-03-16T14:56:39.660041286Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subscript... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc001621880>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-rwkvnl" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5", "name": "b71f3c20-bebb-45a9-8de6-fc37671e60f5", "status": "Failed", "startTime": "2026-03-16T14:55:54.425365996Z", "endTime": "2026-03-16T14:56:39.660041286Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-rwkvnl\" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5\",\n \"name\": \"b71f3c20-bebb-45a9-8de6-fc37671e60f5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:55:54.425365996Z\",\n \"endTime\": \"2026-03-16T14:56:39.660041286Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001621860>{ msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-rwkvnl\" to finish creating: GET https://rp.j8015360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b71f3c20-bebb-45a9-8de6-fc37671e60f5\",\n \"name\": \"b71f3c20-bebb-45a9-8de6-fc37671e60f5\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:55:54.425365996Z\",\n \"endTime\": \"2026-03-16T14:56:39.660041286Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-253/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subscript... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:08 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0008ec3c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-cw7t2h" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d15540>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-cw7t2h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d15520>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-cw7t2h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d15500>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-cw7t2h\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0008ec3c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-cw7t2h" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d15540>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-cw7t2h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d15520>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-cw7t2h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d15500>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-cw7t2h\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:08 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000de02e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-nbrntz" in resourcegroup="rg-zstream-upgrade-4-21-m85s2d" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0000df520>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-nbrntz\" in resourcegroup=\"rg-zstream-upgrade-4-21-m85s2d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0000df500>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-nbrntz\" in resourcegroup=\"rg-zstream-upgrade-4-21-m85s2d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0000df4e0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-nbrntz\" in resourcegroup=\"rg-zstream-upgrade-4-21-m85s2d\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000de02e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-nbrntz" in resourcegroup="rg-zstream-upgrade-4-21-m85s2d" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0000df520>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-nbrntz\" in resourcegroup=\"rg-zstream-upgrade-4-21-m85s2d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0000df500>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-nbrntz\" in resourcegroup=\"rg-zstream-upgrade-4-21-m85s2d\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0000df4e0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-nbrntz\" in resourcegroup=\"rg-zstream-upgrade-4-21-m85s2d\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:39:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d06318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-qbf67t" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b8fa20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-qbf67t\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b8fa00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-qbf67t\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b8f9e0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-qbf67t\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d06318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-qbf67t" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b8fa20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-qbf67t\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b8fa00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-qbf67t\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b8f9e0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-qbf67t\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:39:08 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.000s. Expected success, but got an error: <errors.errorString \| 0xc0007e2620>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.000s. Expected success, but got an error: <errors.errorString \| 0xc0007e2620>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", }
11:39:01	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4e77cea	n/a	unknown	failed	#4435 (closed)	1	non-artifact-backed failure (no junit artifacts)
11:38:39	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 0928008 · merge 0928008	n/a	e2e	failed	#4444 (merged)	13	e2e (13) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:39 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f841c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-tmwxnk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001580e20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tmwxnk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001580e00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tmwxnk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001580de0>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tmwxnk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f841c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-tmwxnk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001580e20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tmwxnk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001580e00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tmwxnk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001580de0>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tmwxnk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:39 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f68bd0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-s5r6vp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a179a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-s5r6vp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a17980>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-s5r6vp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a17960>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-s5r6vp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f68bd0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-s5r6vp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a179a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-s5r6vp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a17980>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-s5r6vp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a17960>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-s5r6vp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift 11:38:39 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc0016e9500>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-59scjg" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31274652-8ea0-44d0-a35f-6e706cc45cef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31274652-8ea0-44d0-a35f-6e706cc45cef", "name": "31274652-8ea0-44d0-a35f-6e706cc45cef", "status": "Failed", "startTime": "2026-03-16T12:34:49.655222627Z", "endTime": "2026-03-16T12:35:29.879696907Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-105/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-105/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get network security group 'engineering-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59scjg/providers/Microsoft.Network/networkSecurityGroups/engineering-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-59scjg\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31274652-8ea0-44d0-a35f-6e706cc45cef\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31274652-8ea0-44d0-a35f-6e706cc45cef\",\n \"name\": \"31274652-8ea0-44d0-a35f-6e706cc45cef\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:49.655222627Z\",\n \"endTime\": \"2026-03-16T12:35:29.879696907Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-105/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-105/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'engineering-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59scjg/providers/Microsoft.Network/networkSecurityGroups/engineering-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0016e94e0>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-59scjg\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31274652-8ea0-44d0-a35f-6e706cc45cef\n--------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc0016e9500>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-59scjg" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31274652-8ea0-44d0-a35f-6e706cc45cef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31274652-8ea0-44d0-a35f-6e706cc45cef", "name": "31274652-8ea0-44d0-a35f-6e706cc45cef", "status": "Failed", "startTime": "2026-03-16T12:34:49.655222627Z", "endTime": "2026-03-16T12:35:29.879696907Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-105/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-105/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get network security group 'engineering-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59scjg/providers/Microsoft.Network/networkSecurityGroups/engineering-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-59scjg\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31274652-8ea0-44d0-a35f-6e706cc45cef\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31274652-8ea0-44d0-a35f-6e706cc45cef\",\n \"name\": \"31274652-8ea0-44d0-a35f-6e706cc45cef\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:49.655222627Z\",\n \"endTime\": \"2026-03-16T12:35:29.879696907Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-105/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-105/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'engineering-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-59scjg/providers/Microsoft.Network/networkSecurityGroups/engineering-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0016e94e0>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-59scjg\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31274652-8ea0-44d0-a35f-6e706cc45cef\n--------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:39 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000139880>: failed to create HCP cluster cluster-zstream-4-19-chcq8w: failed waiting for cluster="cluster-zstream-4-19-chcq8w" in resourcegroup="rg-zstream-upgrade-4-19-rr7nj4" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c", "name": "c5b6da78-84dd-4eba-9fbb-595df90fda1c", "status": "Failed", "startTime": "2026-03-16T12:57:40.329484565Z", "endTime": "2026-03-16T12:58:29.778686734Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-chcq8w: failed waiting for cluster=\"cluster-zstream-4-19-chcq8w\" in resourcegroup=\"rg-zstream-upgrade-4-19-rr7nj4\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c\",\n \"name\": \"c5b6da78-84dd-4eba-9fbb-595df90fda1c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:40.329484565Z\",\n \"endTime\": \"2026-03-16T12:58:29.778686734Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000139860>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-chcq8w\" in resourcegroup=\"rg-zstream-upgrade-4-19-rr7nj4\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c\",\n \"name\": \"c5b6da78-84dd-4eba-9fbb-595df90fda1c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:40.329484565Z\",\n \"endTime\": \"2026-03-16T12:58:29.778686734Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000139880>: failed to create HCP cluster cluster-zstream-4-19-chcq8w: failed waiting for cluster="cluster-zstream-4-19-chcq8w" in resourcegroup="rg-zstream-upgrade-4-19-rr7nj4" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c", "name": "c5b6da78-84dd-4eba-9fbb-595df90fda1c", "status": "Failed", "startTime": "2026-03-16T12:57:40.329484565Z", "endTime": "2026-03-16T12:58:29.778686734Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-chcq8w: failed waiting for cluster=\"cluster-zstream-4-19-chcq8w\" in resourcegroup=\"rg-zstream-upgrade-4-19-rr7nj4\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c\",\n \"name\": \"c5b6da78-84dd-4eba-9fbb-595df90fda1c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:40.329484565Z\",\n \"endTime\": \"2026-03-16T12:58:29.778686734Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000139860>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-chcq8w\" in resourcegroup=\"rg-zstream-upgrade-4-19-rr7nj4\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b6da78-84dd-4eba-9fbb-595df90fda1c\",\n \"name\": \"c5b6da78-84dd-4eba-9fbb-595df90fda1c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:40.329484565Z\",\n \"endTime\": \"2026-03-16T12:58:29.778686734Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:39 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc00151c900>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-mprsps" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba", "name": "8cc48644-1c06-4e82-89c8-139352ede7ba", "status": "Failed", "startTime": "2026-03-16T12:57:42.093796528Z", "endTime": "2026-03-16T12:58:29.776083252Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-mprsps\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba\",\n \"name\": \"8cc48644-1c06-4e82-89c8-139352ede7ba\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:42.093796528Z\",\n \"endTime\": \"2026-03-16T12:58:29.776083252Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00151c8e0>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-mprsps\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba\",\n \"name\": \"8cc48644-1c06-4e82-89c8-139352ede7ba\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:42.093796528Z\",\n \"endTime\": \"2026-03-16T12:58:29.776083252Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc00151c900>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-mprsps" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba", "name": "8cc48644-1c06-4e82-89c8-139352ede7ba", "status": "Failed", "startTime": "2026-03-16T12:57:42.093796528Z", "endTime": "2026-03-16T12:58:29.776083252Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-mprsps\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba\",\n \"name\": \"8cc48644-1c06-4e82-89c8-139352ede7ba\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:42.093796528Z\",\n \"endTime\": \"2026-03-16T12:58:29.776083252Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00151c8e0>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-mprsps\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cc48644-1c06-4e82-89c8-139352ede7ba\",\n \"name\": \"8cc48644-1c06-4e82-89c8-139352ede7ba\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:42.093796528Z\",\n \"endTime\": \"2026-03-16T12:58:29.776083252Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-46/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:39 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010d2408>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-flqft7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00103b140>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-flqft7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00103b120>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-flqft7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00103b100>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-flqft7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010d2408>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-flqft7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00103b140>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-flqft7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00103b120>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-flqft7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00103b100>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-flqft7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:39 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f14240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-fwpz2w" in resourcegroup="rg-zstream-upgrade-4-21-qn75gn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d921a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-fwpz2w\" in resourcegroup=\"rg-zstream-upgrade-4-21-qn75gn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d92160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-fwpz2w\" in resourcegroup=\"rg-zstream-upgrade-4-21-qn75gn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d92140>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-fwpz2w\" in resourcegroup=\"rg-zstream-upgrade-4-21-qn75gn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f14240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-fwpz2w" in resourcegroup="rg-zstream-upgrade-4-21-qn75gn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000d921a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-fwpz2w\" in resourcegroup=\"rg-zstream-upgrade-4-21-qn75gn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d92160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-fwpz2w\" in resourcegroup=\"rg-zstream-upgrade-4-21-qn75gn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000d92140>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-fwpz2w\" in resourcegroup=\"rg-zstream-upgrade-4-21-qn75gn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:39 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00137c240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7kh2bf" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0010b1120>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7kh2bf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0010b1100>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7kh2bf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0010b10e0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7kh2bf\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00137c240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7kh2bf" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0010b1120>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7kh2bf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0010b1100>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7kh2bf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0010b10e0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7kh2bf\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:39 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc001315380>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-b99fd4" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd", "name": "4da2cde2-7773-4b77-9b00-7f6387ea43fd", "status": "Failed", "startTime": "2026-03-16T12:57:41.183698805Z", "endTime": "2026-03-16T12:58:29.788885543Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-b99fd4\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd\",\n \"name\": \"4da2cde2-7773-4b77-9b00-7f6387ea43fd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:41.183698805Z\",\n \"endTime\": \"2026-03-16T12:58:29.788885543Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001315360>{ msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-b99fd4\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd\",\n \"name\": \"4da2cde2-7773-4b77-9b00-7f6387ea43fd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:41.183698805Z\",\n \"endTime\": \"2026-03-16T12:58:29.788885543Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subscript... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc001315380>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-b99fd4" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd", "name": "4da2cde2-7773-4b77-9b00-7f6387ea43fd", "status": "Failed", "startTime": "2026-03-16T12:57:41.183698805Z", "endTime": "2026-03-16T12:58:29.788885543Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-b99fd4\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd\",\n \"name\": \"4da2cde2-7773-4b77-9b00-7f6387ea43fd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:41.183698805Z\",\n \"endTime\": \"2026-03-16T12:58:29.788885543Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001315360>{ msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-b99fd4\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4da2cde2-7773-4b77-9b00-7f6387ea43fd\",\n \"name\": \"4da2cde2-7773-4b77-9b00-7f6387ea43fd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:57:41.183698805Z\",\n \"endTime\": \"2026-03-16T12:58:29.788885543Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subscript... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:39 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00136c1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-lctbf2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001418ec0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-lctbf2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001418ea0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-lctbf2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001418e80>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-lctbf2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00136c1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-lctbf2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001418ec0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-lctbf2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001418ea0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-lctbf2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001418e80>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-lctbf2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:39 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0013b2228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-4dk9v7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007bdbe0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-4dk9v7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007bdbc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-4dk9v7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007bdba0>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-4dk9v7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0013b2228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-4dk9v7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007bdbe0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-4dk9v7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007bdbc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-4dk9v7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007bdba0>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-4dk9v7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:39 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ca8438>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-689h6l" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0011a1140>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-689h6l\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011a1120>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-689h6l\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011a1100>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-689h6l\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ca8438>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-689h6l" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0011a1140>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-689h6l\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011a1120>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-689h6l\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011a1100>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-689h6l\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:39 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc00169b6a0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-ctj74c" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7", "name": "5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7", "status": "Failed", "startTime": "2026-03-16T12:35:38.355734247Z", "endTime": "2026-03-16T12:36:31.750918101Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-ctj74c\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\",\n \"name\": \"5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:35:38.355734247Z\",\n \"endTime\": \"2026-03-16T12:36:31.750918101Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00169b680>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-ctj74c\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\",\n \"name\": \"5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:35:38.355734247Z\",\n \"endTime\": \"2026-03-16T12:36:31.750918101Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc00169b6a0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-ctj74c" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7", "name": "5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7", "status": "Failed", "startTime": "2026-03-16T12:35:38.355734247Z", "endTime": "2026-03-16T12:36:31.750918101Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-ctj74c\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\",\n \"name\": \"5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:35:38.355734247Z\",\n \"endTime\": \"2026-03-16T12:36:31.750918101Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00169b680>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-ctj74c\" to finish creating: GET https://rp.j3253632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\",\n \"name\": \"5339c1b7-6a5a-4939-b1c7-d8c913fdb0e7\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:35:38.355734247Z\",\n \"endTime\": \"2026-03-16T12:36:31.750918101Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-50/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
11:38:31	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 5ed9c72 · merge 1d16d50	n/a	e2e	failed	⚠#4446 (merged)	13	e2e (13) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000a488b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-jj6k4s" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000507ca0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-jj6k4s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000507c80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-jj6k4s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000507c60>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-jj6k4s\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000a488b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-jj6k4s" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000507ca0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-jj6k4s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000507c80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-jj6k4s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000507c60>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-jj6k4s\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:31 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00176a210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-c8v8vx" in resourcegroup="rg-zstream-upgrade-4-21-6h9w42" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00014bd20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-c8v8vx\" in resourcegroup=\"rg-zstream-upgrade-4-21-6h9w42\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00014bd00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-c8v8vx\" in resourcegroup=\"rg-zstream-upgrade-4-21-6h9w42\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00014bce0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-c8v8vx\" in resourcegroup=\"rg-zstream-upgrade-4-21-6h9w42\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00176a210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-c8v8vx" in resourcegroup="rg-zstream-upgrade-4-21-6h9w42" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00014bd20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-c8v8vx\" in resourcegroup=\"rg-zstream-upgrade-4-21-6h9w42\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00014bd00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-c8v8vx\" in resourcegroup=\"rg-zstream-upgrade-4-21-6h9w42\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00014bce0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-c8v8vx\" in resourcegroup=\"rg-zstream-upgrade-4-21-6h9w42\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:31 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001130120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-wtq2pg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a04fc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wtq2pg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a04f80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wtq2pg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a04f60>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wtq2pg\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001130120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-wtq2pg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a04fc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wtq2pg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a04f80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wtq2pg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a04f60>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wtq2pg\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:38:31 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.005s. Expected success, but got an error: <errors.errorString \| 0xc00075a4a0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.005s. Expected success, but got an error: <errors.errorString \| 0xc00075a4a0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:31 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00173e1f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-2df2bg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ed2ea0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-2df2bg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ed2e80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-2df2bg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ed2e60>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-2df2bg\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00173e1f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-2df2bg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ed2ea0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-2df2bg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ed2e80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-2df2bg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ed2e60>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-2df2bg\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:31 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc00080e580>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-xfg9dt" to finish creating: GET https://rp.j2633728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad", "name": "aea717e8-25cf-4ca9-bda1-ba9361a650ad", "status": "Failed", "startTime": "2026-03-16T15:02:20.561460313Z", "endTime": "2026-03-16T15:03:24.977979414Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-xfg9dt\" to finish creating: GET https://rp.j2633728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad\",\n \"name\": \"aea717e8-25cf-4ca9-bda1-ba9361a650ad\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T15:02:20.561460313Z\",\n \"endTime\": \"2026-03-16T15:03:24.977979414Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00080e540>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-xfg9dt\" to finish creating: GET https://rp.j2633728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad\",\n \"name\": \"aea717e8-25cf-4ca9-bda1-ba9361a650ad\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T15:02:20.561460313Z\",\n \"endTime\": \"2026-03-16T15:03:24.977979414Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc00080e580>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-xfg9dt" to finish creating: GET https://rp.j2633728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad", "name": "aea717e8-25cf-4ca9-bda1-ba9361a650ad", "status": "Failed", "startTime": "2026-03-16T15:02:20.561460313Z", "endTime": "2026-03-16T15:03:24.977979414Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-xfg9dt\" to finish creating: GET https://rp.j2633728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad\",\n \"name\": \"aea717e8-25cf-4ca9-bda1-ba9361a650ad\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T15:02:20.561460313Z\",\n \"endTime\": \"2026-03-16T15:03:24.977979414Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00080e540>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-xfg9dt\" to finish creating: GET https://rp.j2633728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aea717e8-25cf-4ca9-bda1-ba9361a650ad\",\n \"name\": \"aea717e8-25cf-4ca9-bda1-ba9361a650ad\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T15:02:20.561460313Z\",\n \"endTime\": \"2026-03-16T15:03:24.977979414Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-144/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:31 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00097c6f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-5ktksf" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ab75c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5ktksf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ab75a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5ktksf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ab7580>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5ktksf\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00097c6f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-5ktksf" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ab75c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5ktksf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ab75a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5ktksf\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ab7580>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5ktksf\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:31 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016620c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-n8ww9v" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0012873c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-n8ww9v\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0012873a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-n8ww9v\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001287380>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-n8ww9v\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016620c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-n8ww9v" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0012873c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-n8ww9v\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0012873a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-n8ww9v\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001287380>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-n8ww9v\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fdc3f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-kmmt9r" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00088a260>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-kmmt9r\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088a240>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-kmmt9r\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088a200>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-kmmt9r\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fdc3f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-kmmt9r" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00088a260>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-kmmt9r\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088a240>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-kmmt9r\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088a200>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-kmmt9r\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:31 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d38270>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-jbs926" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000fe4820>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-jbs926\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fe47e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-jbs926\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013f9520>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-jbs926\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d38270>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-jbs926" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000fe4820>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-jbs926\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fe47e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-jbs926\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0013f9520>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-jbs926\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e64258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-q842nj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b2fb60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-q842nj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b2fb20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-q842nj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b2fae0>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-q842nj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e64258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-q842nj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b2fb60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-q842nj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b2fb20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-q842nj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b2fae0>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-q842nj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code 429; detail message Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds.; provider Microsoft.ManagedIdentity 11:38:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapError \| 0xc000b35bc0>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-2ttk57" to finish creating: GET https://rp.j2633728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6031911-24e5-4ffa-a97f-ee993d430b06 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6031911-24e5-4ffa-a97f-ee993d430b06", "name": "a6031911-24e5-4ffa-a97f-ee993d430b06", "status": "Failed", "startTime": "2026-03-16T14:28:14.441998363Z", "endTime": "2026-03-16T14:29:08.761109691Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-121/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-121/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-2ttk57\" to finish creating: GET https://rp.j2633728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6031911-24e5-4ffa-a97f-ee993d430b06\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6031911-24e5-4ffa-a97f-ee993d430b06\",\n \"name\": \"a6031911-24e5-4ffa-a97f-ee993d430b06\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:28:14.441998363Z\",\n \"endTime\": \"2026-03-16T14:29:08.761109691Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-121/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-121/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012455f0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "be961142-a154-4bed-8f44-db4c4e95ae55", ], "Date": [ "Mon, 16 Mar 2026 14:29:14 GMT", ], "Content-Length": ["1382"], "X-Envoy-Upstream-Service-Time": ["53"], }, Body: <exported.nopClosingBytesReader \| 0xc000b35600>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 97, 54, 48, 51, 49, 57, 49, 49, 45, 50, 52, 101, 53, 45, 52, 102, 102, 97, 45, 97, 57, 55, 102, 45, 101, 101, 57, 57, 51, 100, 52, 51, 48, 98, 48, 54, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 97, 54, 48, 51, 49, 57, 49, 49, 45, 50, 52, 101, 53, 45, 52, 102, 102, 97, 45, 97, 57, 55, 102, 45, 101, 101, 57, 57, 51, 100, 52, 51, 48, 98, 48, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapError \| 0xc000b35bc0>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-2ttk57" to finish creating: GET https://rp.j2633728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6031911-24e5-4ffa-a97f-ee993d430b06 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6031911-24e5-4ffa-a97f-ee993d430b06", "name": "a6031911-24e5-4ffa-a97f-ee993d430b06", "status": "Failed", "startTime": "2026-03-16T14:28:14.441998363Z", "endTime": "2026-03-16T14:29:08.761109691Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-121/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-121/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-2ttk57\" to finish creating: GET https://rp.j2633728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6031911-24e5-4ffa-a97f-ee993d430b06\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6031911-24e5-4ffa-a97f-ee993d430b06\",\n \"name\": \"a6031911-24e5-4ffa-a97f-ee993d430b06\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:28:14.441998363Z\",\n \"endTime\": \"2026-03-16T14:29:08.761109691Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-121/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-121/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 5 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012455f0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "be961142-a154-4bed-8f44-db4c4e95ae55", ], "Date": [ "Mon, 16 Mar 2026 14:29:14 GMT", ], "Content-Length": ["1382"], "X-Envoy-Upstream-Service-Time": ["53"], }, Body: <exported.nopClosingBytesReader \| 0xc000b35600>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 97, 54, 48, 51, 49, 57, 49, 49, 45, 50, 52, 101, 53, 45, 52, 102, 102, 97, 45, 97, 57, 55, 102, 45, 101, 101, 57, 57, 51, 100, 52, 51, 48, 98, 48, 54, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 97, 54, 48, 51, 49, 57, 49, 49, 45, 50, 52, 101, 53, 45, 52, 102, 102, 97, 45, 97, 57, 55, 102, 45, 101, 101, 57, 57, 51, 100, 52, 51, 48, 98, 48, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:31 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016e22a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fk5jp8" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00071b960>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fk5jp8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00071b940>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fk5jp8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00071b920>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fk5jp8\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016e22a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fk5jp8" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00071b960>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fk5jp8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00071b940>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fk5jp8\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00071b920>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fk5jp8\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:38:16	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head cb1d0d4 · merge cb1d0d4	n/a	e2e	failed	★#4450 (merged)	8	e2e (8) ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapError \| 0xc000597540>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-pdhhqw" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19", "name": "3125e7dd-2e11-4b1a-a107-ac55289b8b19", "status": "Failed", "startTime": "2026-03-16T13:09:12.515669684Z", "endTime": "2026-03-16T13:10:09.05502246Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-pdhhqw\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19\",\n \"name\": \"3125e7dd-2e11-4b1a-a107-ac55289b8b19\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:12.515669684Z\",\n \"endTime\": \"2026-03-16T13:10:09.05502246Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000597520>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-pdhhqw\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19\",\n \"name\": \"3125e7dd-2e11-4b1a-a107-ac55289b8b19\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:12.515669684Z\",\n \"endTime\": \"2026-03-16T13:10:09.05502246Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapError \| 0xc000597540>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-pdhhqw" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19", "name": "3125e7dd-2e11-4b1a-a107-ac55289b8b19", "status": "Failed", "startTime": "2026-03-16T13:09:12.515669684Z", "endTime": "2026-03-16T13:10:09.05502246Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-pdhhqw\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19\",\n \"name\": \"3125e7dd-2e11-4b1a-a107-ac55289b8b19\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:12.515669684Z\",\n \"endTime\": \"2026-03-16T13:10:09.05502246Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000597520>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-pdhhqw\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3125e7dd-2e11-4b1a-a107-ac55289b8b19\",\n \"name\": \"3125e7dd-2e11-4b1a-a107-ac55289b8b19\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:12.515669684Z\",\n \"endTime\": \"2026-03-16T13:10:09.05502246Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-297/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:16 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapError \| 0xc000921960>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mcrpkg" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823", "name": "f6cd8c6f-2f9a-46ff-8baa-01ea33049823", "status": "Failed", "startTime": "2026-03-16T13:09:25.554283148Z", "endTime": "2026-03-16T13:10:09.075846622Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-mcrpkg\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823\",\n \"name\": \"f6cd8c6f-2f9a-46ff-8baa-01ea33049823\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:25.554283148Z\",\n \"endTime\": \"2026-03-16T13:10:09.075846622Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000921920>{ msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-mcrpkg\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823\",\n \"name\": \"f6cd8c6f-2f9a-46ff-8baa-01ea33049823\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:25.554283148Z\",\n \"endTime\": \"2026-03-16T13:10:09.075846622Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapError \| 0xc000921960>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-mcrpkg" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823", "name": "f6cd8c6f-2f9a-46ff-8baa-01ea33049823", "status": "Failed", "startTime": "2026-03-16T13:09:25.554283148Z", "endTime": "2026-03-16T13:10:09.075846622Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-mcrpkg\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823\",\n \"name\": \"f6cd8c6f-2f9a-46ff-8baa-01ea33049823\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:25.554283148Z\",\n \"endTime\": \"2026-03-16T13:10:09.075846622Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000921920>{ msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-mcrpkg\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6cd8c6f-2f9a-46ff-8baa-01ea33049823\",\n \"name\": \"f6cd8c6f-2f9a-46ff-8baa-01ea33049823\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:25.554283148Z\",\n \"endTime\": \"2026-03-16T13:10:09.075846622Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-100/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:16 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error: <fmt.wrapError \| 0xc0010ca540>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-6bxc98" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b", "name": "02ff7b81-7cfd-4397-963c-dfb697e5c84b", "status": "Failed", "startTime": "2026-03-16T13:09:22.548426273Z", "endTime": "2026-03-16T13:10:09.038847011Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-6bxc98\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b\",\n \"name\": \"02ff7b81-7cfd-4397-963c-dfb697e5c84b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:22.548426273Z\",\n \"endTime\": \"2026-03-16T13:10:09.038847011Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010ca520>{ msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-6bxc98\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b\",\n \"name\": \"02ff7b81-7cfd-4397-963c-dfb697e5c84b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:22.548426273Z\",\n \"endTime\": \"2026-03-16T13:10:09.038847011Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error: <fmt.wrapError \| 0xc0010ca540>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-6bxc98" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b", "name": "02ff7b81-7cfd-4397-963c-dfb697e5c84b", "status": "Failed", "startTime": "2026-03-16T13:09:22.548426273Z", "endTime": "2026-03-16T13:10:09.038847011Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-6bxc98\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b\",\n \"name\": \"02ff7b81-7cfd-4397-963c-dfb697e5c84b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:22.548426273Z\",\n \"endTime\": \"2026-03-16T13:10:09.038847011Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010ca520>{ msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-6bxc98\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02ff7b81-7cfd-4397-963c-dfb697e5c84b\",\n \"name\": \"02ff7b81-7cfd-4397-963c-dfb697e5c84b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:22.548426273Z\",\n \"endTime\": \"2026-03-16T13:10:09.038847011Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-5/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift 11:38:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc000b28c20>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-gjc7zg" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/635b1c05-4f04-4ea3-8f1c-3caef8df7984 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/635b1c05-4f04-4ea3-8f1c-3caef8df7984", "name": "635b1c05-4f04-4ea3-8f1c-3caef8df7984", "status": "Failed", "startTime": "2026-03-16T13:09:13.599864638Z", "endTime": "2026-03-16T13:10:08.97808366Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-gjc7zg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-64/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-64/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-gjc7zg\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/635b1c05-4f04-4ea3-8f1c-3caef8df7984\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/635b1c05-4f04-4ea3-8f1c-3caef8df7984\",\n \"name\": \"635b1c05-4f04-4ea3-8f1c-3caef8df7984\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:13.599864638Z\",\n \"endTime\": \"2026-03-16T13:10:08.97808366Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-gjc7zg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-64/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-64/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b28c00>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-gjc7zg\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/635b1c05-4f04-4ea3-8f1c-3ca... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc000b28c20>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-gjc7zg" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/635b1c05-4f04-4ea3-8f1c-3caef8df7984 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/635b1c05-4f04-4ea3-8f1c-3caef8df7984", "name": "635b1c05-4f04-4ea3-8f1c-3caef8df7984", "status": "Failed", "startTime": "2026-03-16T13:09:13.599864638Z", "endTime": "2026-03-16T13:10:08.97808366Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-gjc7zg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-64/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-64/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-gjc7zg\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/635b1c05-4f04-4ea3-8f1c-3caef8df7984\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/635b1c05-4f04-4ea3-8f1c-3caef8df7984\",\n \"name\": \"635b1c05-4f04-4ea3-8f1c-3caef8df7984\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:13.599864638Z\",\n \"endTime\": \"2026-03-16T13:10:08.97808366Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-gjc7zg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-64/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-64/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b28c00>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-gjc7zg\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/635b1c05-4f04-4ea3-8f1c-3ca... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:16 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000ea58e0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-4f47jw" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd", "name": "657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd", "status": "Failed", "startTime": "2026-03-16T13:09:22.698540737Z", "endTime": "2026-03-16T13:10:09.035865816Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-4f47jw\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\",\n \"name\": \"657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:22.698540737Z\",\n \"endTime\": \"2026-03-16T13:10:09.035865816Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000ea58c0>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-4f47jw\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\",\n \"name\": \"657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:22.698540737Z\",\n \"endTime\": \"2026-03-16T13:10:09.035865816Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000ea58e0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-4f47jw" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd", "name": "657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd", "status": "Failed", "startTime": "2026-03-16T13:09:22.698540737Z", "endTime": "2026-03-16T13:10:09.035865816Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-4f47jw\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\",\n \"name\": \"657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:22.698540737Z\",\n \"endTime\": \"2026-03-16T13:10:09.035865816Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000ea58c0>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-4f47jw\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\",\n \"name\": \"657687f8-0b2a-4bf5-98a8-7b0b31b9e3dd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:22.698540737Z\",\n \"endTime\": \"2026-03-16T13:10:09.035865816Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-62/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:16 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc000a26580>: failed to create HCP cluster ea-listv7w2gg: failed waiting for cluster="ea-listv7w2gg" in resourcegroup="ea-list-rx6tlz" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c", "name": "9cea25ca-7954-433a-ae78-8d863b6ed26c", "status": "Failed", "startTime": "2026-03-16T13:09:21.953309402Z", "endTime": "2026-03-16T13:10:09.037665494Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listv7w2gg: failed waiting for cluster=\"ea-listv7w2gg\" in resourcegroup=\"ea-list-rx6tlz\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c\",\n \"name\": \"9cea25ca-7954-433a-ae78-8d863b6ed26c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:21.953309402Z\",\n \"endTime\": \"2026-03-16T13:10:09.037665494Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a26560>{ msg: "failed waiting for cluster=\"ea-listv7w2gg\" in resourcegroup=\"ea-list-rx6tlz\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c\",\n \"name\": \"9cea25ca-7954-433a-ae78-8d863b6ed26c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:21.953309402Z\",\n \"endTime\": \"2026-03-16T13:10:09.037665494Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"cod... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc000a26580>: failed to create HCP cluster ea-listv7w2gg: failed waiting for cluster="ea-listv7w2gg" in resourcegroup="ea-list-rx6tlz" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c", "name": "9cea25ca-7954-433a-ae78-8d863b6ed26c", "status": "Failed", "startTime": "2026-03-16T13:09:21.953309402Z", "endTime": "2026-03-16T13:10:09.037665494Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listv7w2gg: failed waiting for cluster=\"ea-listv7w2gg\" in resourcegroup=\"ea-list-rx6tlz\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c\",\n \"name\": \"9cea25ca-7954-433a-ae78-8d863b6ed26c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:21.953309402Z\",\n \"endTime\": \"2026-03-16T13:10:09.037665494Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a26560>{ msg: "failed waiting for cluster=\"ea-listv7w2gg\" in resourcegroup=\"ea-list-rx6tlz\" to finish creating: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9cea25ca-7954-433a-ae78-8d863b6ed26c\",\n \"name\": \"9cea25ca-7954-433a-ae78-8d863b6ed26c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T13:09:21.953309402Z\",\n \"endTime\": \"2026-03-16T13:10:09.037665494Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-33/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"cod... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:38:16 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc0008e9650>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s. Expected success, but got an error: <errors.errorString \| 0xc0008e9650>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:16 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:135]: Unexpected error: <exported.ResponseError \| 0xc0015da540>: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b3f846-6506-4eda-9015-7af19755c284 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b3f846-6506-4eda-9015-7af19755c284", "name": "c5b3f846-6506-4eda-9015-7af19755c284", "status": "Failed", "startTime": "2026-03-16T13:10:10.437965568Z", "endTime": "2026-03-16T13:10:54.695097893Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-61/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-61/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "12303f11-8b3e-4d5f-affd-650a453fcf08", ], "Date": [ "Mon, 16 Mar 2026 13:20:24 GMT", ], "Content-Length": ["1573"], "X-Envoy-Upstream-Service-Time": ["9"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000784660>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 99, 53, 98, 51, 102, 56, 52, 54, 45, 54, 53, 48, 54, 45, 52, 101, 100, 97, 45, 57, 48, 49, 53, 45, 55, 97, 102, 49, 57, 55, 53, 53, 99, 50, 56, 52, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 99, 53, 98, 51, 102, 56, 52, 54, 45, 54, 53, 48, 54, 45, 52, 101, 100, 97, 45, 57, 48, 49, 53, 45, 55, 97, 102, 49, 57, 55, 53, 53, 99, 50, 56, 52, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 51, 58, 49, 48, 58, 49, 48, 46, 52, 51, 55, 57, 54, 53, 53, 54, 56, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 51, 58, 49, 48, 58, 53, 52, 46, 54, 57, 53, 48, 57, 55, 56, 57, 51, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 54, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 99, 108, 111, 117, 100, 45, 110, 101, 116, 119, 111, 114, 107, 45, 99, 111, 110, 102, 105, 103, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:135]: Unexpected error: <exported.ResponseError \| 0xc0015da540>: GET https://rp.j8596480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b3f846-6506-4eda-9015-7af19755c284 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c5b3f846-6506-4eda-9015-7af19755c284", "name": "c5b3f846-6506-4eda-9015-7af19755c284", "status": "Failed", "startTime": "2026-03-16T13:10:10.437965568Z", "endTime": "2026-03-16T13:10:54.695097893Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-61/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-61/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "12303f11-8b3e-4d5f-affd-650a453fcf08", ], "Date": [ "Mon, 16 Mar 2026 13:20:24 GMT", ], "Content-Length": ["1573"], "X-Envoy-Upstream-Service-Time": ["9"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000784660>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 99, 53, 98, 51, 102, 56, 52, 54, 45, 54, 53, 48, 54, 45, 52, 101, 100, 97, 45, 57, 48, 49, 53, 45, 55, 97, 102, 49, 57, 55, 53, 53, 99, 50, 56, 52, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 99, 53, 98, 51, 102, 56, 52, 54, 45, 54, 53, 48, 54, 45, 52, 101, 100, 97, 45, 57, 48, 49, 53, 45, 55, 97, 102, 49, 57, 55, 53, 53, 99, 50, 56, 52, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 51, 58, 49, 48, 58, 49, 48, 46, 52, 51, 55, 57, 54, 53, 53, 54, 56, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 51, 58, 49, 48, 58, 53, 52, 46, 54, 57, 53, 48, 57, 55, 56, 57, 51, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 54, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 99, 108, 111, 117, 100, 45, 110, 101, 116, 119, 111, 114, 107, 45, 99, 111, 110, 102, 105, 103, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
11:38:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 7406646 · merge 7406646	n/a	e2e	failed	★#4453 (merged)	11	e2e (11) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:08 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000c420c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listpwbm95" in resourcegroup="ea-list-z65xhn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000aa1440>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listpwbm95\" in resourcegroup=\"ea-list-z65xhn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000aa1420>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listpwbm95\" in resourcegroup=\"ea-list-z65xhn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000aa1400>{ msg: "failed waiting for hcpCluster=\"ea-listpwbm95\" in resourcegroup=\"ea-list-z65xhn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000c420c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listpwbm95" in resourcegroup="ea-list-z65xhn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000aa1440>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listpwbm95\" in resourcegroup=\"ea-list-z65xhn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000aa1420>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listpwbm95\" in resourcegroup=\"ea-list-z65xhn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000aa1400>{ msg: "failed waiting for hcpCluster=\"ea-listpwbm95\" in resourcegroup=\"ea-list-z65xhn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:08 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e16240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-qm5msk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007d5f20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-qm5msk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007d5f00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-qm5msk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007d5ee0>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-qm5msk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e16240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-qm5msk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007d5f20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-qm5msk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007d5f00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-qm5msk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007d5ee0>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-qm5msk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:08 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010a0378>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-5xj8sr" in resourcegroup="rg-zstream-upgrade-4-20-77qhqq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00104cfa0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-5xj8sr\" in resourcegroup=\"rg-zstream-upgrade-4-20-77qhqq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00104cf80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-5xj8sr\" in resourcegroup=\"rg-zstream-upgrade-4-20-77qhqq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00104cf60>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-5xj8sr\" in resourcegroup=\"rg-zstream-upgrade-4-20-77qhqq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010a0378>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-5xj8sr" in resourcegroup="rg-zstream-upgrade-4-20-77qhqq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00104cfa0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-5xj8sr\" in resourcegroup=\"rg-zstream-upgrade-4-20-77qhqq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00104cf80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-5xj8sr\" in resourcegroup=\"rg-zstream-upgrade-4-20-77qhqq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00104cf60>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-5xj8sr\" in resourcegroup=\"rg-zstream-upgrade-4-20-77qhqq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:08 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000012138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-lxsgpq" in resourcegroup="admin-credential-lifecycle-test-57qnvt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000980ca0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-lxsgpq\" in resourcegroup=\"admin-credential-lifecycle-test-57qnvt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000980c80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-lxsgpq\" in resourcegroup=\"admin-credential-lifecycle-test-57qnvt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000980c60>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-lxsgpq\" in resourcegroup=\"admin-credential-lifecycle-test-57qnvt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000012138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-lxsgpq" in resourcegroup="admin-credential-lifecycle-test-57qnvt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000980ca0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-lxsgpq\" in resourcegroup=\"admin-credential-lifecycle-test-57qnvt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000980c80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-lxsgpq\" in resourcegroup=\"admin-credential-lifecycle-test-57qnvt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000980c60>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-lxsgpq\" in resourcegroup=\"admin-credential-lifecycle-test-57qnvt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:08 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f2e228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-m9tz4j" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007acf80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-m9tz4j\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007acf60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-m9tz4j\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007acf40>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-m9tz4j\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f2e228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-m9tz4j" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007acf80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-m9tz4j\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007acf60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-m9tz4j\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007acf40>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-m9tz4j\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:08 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fdbf50>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qzw5x5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007365a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-qzw5x5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000736580>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-qzw5x5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000736560>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-qzw5x5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fdbf50>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qzw5x5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007365a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-qzw5x5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000736580>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-qzw5x5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000736560>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-qzw5x5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:08 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc001112a40>: failed to create HCP cluster cluster-zstream-4-19-qmv88n: failed waiting for cluster="cluster-zstream-4-19-qmv88n" in resourcegroup="rg-zstream-upgrade-4-19-6vfmfp" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928", "name": "cffe6b5d-2234-4df7-a21f-204b7b5db928", "status": "Failed", "startTime": "2026-03-16T14:56:50.633649849Z", "endTime": "2026-03-16T14:57:37.094827871Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-qmv88n: failed waiting for cluster=\"cluster-zstream-4-19-qmv88n\" in resourcegroup=\"rg-zstream-upgrade-4-19-6vfmfp\" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928\",\n \"name\": \"cffe6b5d-2234-4df7-a21f-204b7b5db928\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:56:50.633649849Z\",\n \"endTime\": \"2026-03-16T14:57:37.094827871Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001112a20>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-qmv88n\" in resourcegroup=\"rg-zstream-upgrade-4-19-6vfmfp\" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928\",\n \"name\": \"cffe6b5d-2234-4df7-a21f-204b7b5db928\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:56:50.633649849Z\",\n \"endTime\": \"2026-03-16T14:57:37.094827871Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc001112a40>: failed to create HCP cluster cluster-zstream-4-19-qmv88n: failed waiting for cluster="cluster-zstream-4-19-qmv88n" in resourcegroup="rg-zstream-upgrade-4-19-6vfmfp" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928", "name": "cffe6b5d-2234-4df7-a21f-204b7b5db928", "status": "Failed", "startTime": "2026-03-16T14:56:50.633649849Z", "endTime": "2026-03-16T14:57:37.094827871Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-qmv88n: failed waiting for cluster=\"cluster-zstream-4-19-qmv88n\" in resourcegroup=\"rg-zstream-upgrade-4-19-6vfmfp\" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928\",\n \"name\": \"cffe6b5d-2234-4df7-a21f-204b7b5db928\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:56:50.633649849Z\",\n \"endTime\": \"2026-03-16T14:57:37.094827871Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001112a20>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-qmv88n\" in resourcegroup=\"rg-zstream-upgrade-4-19-6vfmfp\" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cffe6b5d-2234-4df7-a21f-204b7b5db928\",\n \"name\": \"cffe6b5d-2234-4df7-a21f-204b7b5db928\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:56:50.633649849Z\",\n \"endTime\": \"2026-03-16T14:57:37.094827871Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-69/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:08 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e22300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-d5qf5c" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009237a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-d5qf5c\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000923600>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-d5qf5c\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009235e0>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-d5qf5c\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e22300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-d5qf5c" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009237a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-d5qf5c\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000923600>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-d5qf5c\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009235e0>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-d5qf5c\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fec390>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-2m2fvl" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a33380>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m2fvl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a33360>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m2fvl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a33340>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m2fvl\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fec390>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-2m2fvl" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a33380>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m2fvl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a33360>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m2fvl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a33340>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-2m2fvl\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:38:08 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:151]: Unexpected error: <fmt.wrapError \| 0xc000b6e260>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster3-bxsgwq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-bxsgwq/operationStatuses/08584279348715665067 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-m97f8n/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-bxsgwq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-bxsgwq/operationStatuses/08584279348715665067\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-m97f8n/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b6e240>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-bxsgwq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-bxsgwq/operationStatuses/08584279348715665067\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-m97f8n/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Ple... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:151]: Unexpected error: <fmt.wrapError \| 0xc000b6e260>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster3-bxsgwq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-bxsgwq/operationStatuses/08584279348715665067 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-m97f8n/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-bxsgwq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-bxsgwq/operationStatuses/08584279348715665067\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-m97f8n/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b6e240>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster3-bxsgwq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster3-bxsgwq/operationStatuses/08584279348715665067\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-m97f8n/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-igpwm4vimfr3y\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Ple... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:08 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc00093a7e0>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-627pgj" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb", "name": "7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb", "status": "Failed", "startTime": "2026-03-16T14:32:49.565417077Z", "endTime": "2026-03-16T14:33:53.85749565Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-627pgj\" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\",\n \"name\": \"7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:32:49.565417077Z\",\n \"endTime\": \"2026-03-16T14:33:53.85749565Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00093a7c0>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-627pgj\" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\",\n \"name\": \"7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:32:49.565417077Z\",\n \"endTime\": \"2026-03-16T14:33:53.85749565Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc00093a7e0>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-627pgj" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb", "name": "7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb", "status": "Failed", "startTime": "2026-03-16T14:32:49.565417077Z", "endTime": "2026-03-16T14:33:53.85749565Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-627pgj\" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\",\n \"name\": \"7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:32:49.565417077Z\",\n \"endTime\": \"2026-03-16T14:33:53.85749565Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00093a7c0>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-627pgj\" to finish creating: GET https://rp.j8971520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\",\n \"name\": \"7084ccd4-bfdb-480b-8ef1-2dcb5788cdcb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:32:49.565417077Z\",\n \"endTime\": \"2026-03-16T14:33:53.85749565Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-132/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
11:38:01	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 38f866a · merge 38f866a	n/a	e2e	failed	★#4454 (merged)	20	e2e (20) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d84258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listptm4dr" in resourcegroup="ea-list-wml2kk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000cf6280>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listptm4dr\" in resourcegroup=\"ea-list-wml2kk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000cf6240>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listptm4dr\" in resourcegroup=\"ea-list-wml2kk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000cf6220>{ msg: "failed waiting for hcpCluster=\"ea-listptm4dr\" in resourcegroup=\"ea-list-wml2kk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d84258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listptm4dr" in resourcegroup="ea-list-wml2kk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000cf6280>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listptm4dr\" in resourcegroup=\"ea-list-wml2kk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000cf6240>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listptm4dr\" in resourcegroup=\"ea-list-wml2kk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000cf6220>{ msg: "failed waiting for hcpCluster=\"ea-listptm4dr\" in resourcegroup=\"ea-list-wml2kk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred client rate limiter Wait returned an error: context deadline exceeded 11:38:01 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc000ffeed0>: context deadline exceeded failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded { errs: [ <context.deadlineExceededError>{}, <fmt.wrapError \| 0xc000fe65e0>{ msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded", err: <fmt.wrapError \| 0xc000fe65c0>{ msg: "client rate limiter Wait returned an error: context deadline exceeded", err: <context.deadlineExceededError>{}, }, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc000ffeed0>: context deadline exceeded failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded { errs: [ <context.deadlineExceededError>{}, <fmt.wrapError \| 0xc000fe65e0>{ msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded", err: <fmt.wrapError \| 0xc000fe65c0>{ msg: "client rate limiter Wait returned an error: context deadline exceeded", err: <context.deadlineExceededError>{}, }, }, ], } occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:38:01 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc0008dc840>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-t27s5z-d2k5jf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-t27s5z-d2k5jf/operationStatuses/08584279347964614527 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-p2xqvm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xruflpgmwocnq\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-t27s5z-d2k5jf\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-t27s5z-d2k5jf/operationStatuses/08584279347964614527\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-p2xqvm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xruflpgmwocnq\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008dc7e0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-t27s5z-d2k5jf\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-t27s5z-d2k5jf/operationStatuses/08584279347964614527\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nER... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc0008dc840>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-t27s5z-d2k5jf" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-t27s5z-d2k5jf/operationStatuses/08584279347964614527 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-p2xqvm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xruflpgmwocnq\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-t27s5z-d2k5jf\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-t27s5z-d2k5jf/operationStatuses/08584279347964614527\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-p2xqvm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xruflpgmwocnq\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008dc7e0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-t27s5z-d2k5jf\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-t27s5z-d2k5jf/operationStatuses/08584279347964614527\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nER... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010c6258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-8mzj6g" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ea2e20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-8mzj6g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ea2e00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-8mzj6g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ea2de0>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-8mzj6g\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010c6258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-8mzj6g" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ea2e20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-8mzj6g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ea2e00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-8mzj6g\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ea2de0>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-8mzj6g\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:38:01 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000777540>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-nsdxc6-fqc8ch" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-nsdxc6-fqc8ch/operationStatuses/08584279348074294009 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9dgkn9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-d22ejdsr2z4k6\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-nsdxc6-fqc8ch\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-nsdxc6-fqc8ch/operationStatuses/08584279348074294009\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9dgkn9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-d22ejdsr2z4k6\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000777540>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-nsdxc6-fqc8ch" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-nsdxc6-fqc8ch/operationStatuses/08584279348074294009 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9dgkn9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-d22ejdsr2z4k6\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-nsdxc6-fqc8ch\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-nsdxc6-fqc8ch/operationStatuses/08584279348074294009\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-9dgkn9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-d22ejdsr2z4k6\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:38:01 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:79]: Unexpected error: <fmt.wrapError \| 0xc000bb19c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-wgq4wx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-wgq4wx/operationStatuses/08584279346171694286 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-skmwld/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-wgq4wx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-wgq4wx/operationStatuses/08584279346171694286\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-skmwld/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:79]: Unexpected error: <fmt.wrapError \| 0xc000bb19c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-wgq4wx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-wgq4wx/operationStatuses/08584279346171694286 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-skmwld/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-wgq4wx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-wgq4wx/operationStatuses/08584279346171694286\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-skmwld/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:01 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc0001da140>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kl2nls" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc", "name": "fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc", "status": "Failed", "startTime": "2026-03-16T14:36:47.922929088Z", "endTime": "2026-03-16T14:37:44.479957319Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-kl2nls\" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\",\n \"name\": \"fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:47.922929088Z\",\n \"endTime\": \"2026-03-16T14:37:44.479957319Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0001da120>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-kl2nls\" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\",\n \"name\": \"fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:47.922929088Z\",\n \"endTime\": \"2026-03-16T14:37:44.479957319Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc0001da140>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kl2nls" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc", "name": "fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc", "status": "Failed", "startTime": "2026-03-16T14:36:47.922929088Z", "endTime": "2026-03-16T14:37:44.479957319Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-kl2nls\" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\",\n \"name\": \"fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:47.922929088Z\",\n \"endTime\": \"2026-03-16T14:37:44.479957319Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0001da120>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-kl2nls\" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\",\n \"name\": \"fc8ee5d3-6bb5-4055-8f4b-9e79f49627cc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:36:47.922929088Z\",\n \"endTime\": \"2026-03-16T14:37:44.479957319Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00157a1c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-mq9qzt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00068cb40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-mq9qzt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00068cb20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-mq9qzt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00068cb00>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-mq9qzt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00157a1c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-mq9qzt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00068cb40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-mq9qzt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00068cb20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-mq9qzt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00068cb00>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-mq9qzt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0011f82e8>: failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-wddhhw" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000989600>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-wddhhw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009895e0>{ msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-wddhhw\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0011f82e8>: failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-wddhhw" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000989600>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-wddhhw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009895e0>{ msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-wddhhw\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0013c01e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-rdwmn6" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000832d60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rdwmn6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000832d40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rdwmn6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000832d20>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rdwmn6\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0013c01e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-rdwmn6" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000832d60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rdwmn6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000832d40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rdwmn6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000832d20>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-rdwmn6\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dd8678>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-7m89cx" in resourcegroup="rg-zstream-upgrade-4-21-rf6mkp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000c9f520>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-7m89cx\" in resourcegroup=\"rg-zstream-upgrade-4-21-rf6mkp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c9f500>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-7m89cx\" in resourcegroup=\"rg-zstream-upgrade-4-21-rf6mkp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c9f4e0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-7m89cx\" in resourcegroup=\"rg-zstream-upgrade-4-21-rf6mkp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dd8678>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-7m89cx" in resourcegroup="rg-zstream-upgrade-4-21-rf6mkp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000c9f520>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-7m89cx\" in resourcegroup=\"rg-zstream-upgrade-4-21-rf6mkp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c9f500>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-7m89cx\" in resourcegroup=\"rg-zstream-upgrade-4-21-rf6mkp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c9f4e0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-7m89cx\" in resourcegroup=\"rg-zstream-upgrade-4-21-rf6mkp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001526240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-scqfhl" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007b3920>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-scqfhl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007b3900>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-scqfhl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007b38e0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-scqfhl\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001526240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-scqfhl" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007b3920>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-scqfhl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007b3900>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-scqfhl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007b38e0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-scqfhl\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d94378>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-fczs4p" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006fd920>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-fczs4p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006fd8e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-fczs4p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006fd8c0>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-fczs4p\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d94378>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-fczs4p" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006fd920>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-fczs4p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006fd8e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-fczs4p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006fd8c0>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-fczs4p\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:38:01 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc000e4c600>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-bjhgrb" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b", "name": "cb56abe4-c630-47aa-9f92-e717ef68f70b", "status": "Failed", "startTime": "2026-03-16T14:37:42.088418326Z", "endTime": "2026-03-16T14:38:25.667213121Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-bjhgrb\" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b\",\n \"name\": \"cb56abe4-c630-47aa-9f92-e717ef68f70b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:37:42.088418326Z\",\n \"endTime\": \"2026-03-16T14:38:25.667213121Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e4c5e0>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-bjhgrb\" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b\",\n \"name\": \"cb56abe4-c630-47aa-9f92-e717ef68f70b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:37:42.088418326Z\",\n \"endTime\": \"2026-03-16T14:38:25.667213121Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc000e4c600>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-bjhgrb" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b", "name": "cb56abe4-c630-47aa-9f92-e717ef68f70b", "status": "Failed", "startTime": "2026-03-16T14:37:42.088418326Z", "endTime": "2026-03-16T14:38:25.667213121Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-bjhgrb\" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b\",\n \"name\": \"cb56abe4-c630-47aa-9f92-e717ef68f70b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:37:42.088418326Z\",\n \"endTime\": \"2026-03-16T14:38:25.667213121Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e4c5e0>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-bjhgrb\" to finish creating: GET https://rp.j7711744.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb56abe4-c630-47aa-9f92-e717ef68f70b\",\n \"name\": \"cb56abe4-c630-47aa-9f92-e717ef68f70b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:37:42.088418326Z\",\n \"endTime\": \"2026-03-16T14:38:25.667213121Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-114/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e9e630>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-k688nh" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0014f5180>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k688nh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014f5160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k688nh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014f5140>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k688nh\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e9e630>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-k688nh" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0014f5180>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k688nh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014f5160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k688nh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0014f5140>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k688nh\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000189140>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-nd6h7p" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000661d20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-nd6h7p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000661d00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-nd6h7p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000661ce0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-nd6h7p\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000189140>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-nd6h7p" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000661d20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-nd6h7p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000661d00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-nd6h7p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000661ce0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-nd6h7p\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:38:01 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc00148c5a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-9gt4qd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-9gt4qd/operationStatuses/08584279346169881662 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-7fn6kk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-9gt4qd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-9gt4qd/operationStatuses/08584279346169881662\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-7fn6kk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc00148c5a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-9gt4qd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-9gt4qd/operationStatuses/08584279346169881662 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-7fn6kk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-9gt4qd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-9gt4qd/operationStatuses/08584279346169881662\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-55/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-7fn6kk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001494108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-cdsw9p" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00088f8e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-cdsw9p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088f8c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-cdsw9p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088f6e0>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-cdsw9p\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001494108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-cdsw9p" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00088f8e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-cdsw9p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088f8c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-cdsw9p\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00088f6e0>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-cdsw9p\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001802270>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-msmfdt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000eaef20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-msmfdt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000eaef00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-msmfdt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000eaeee0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-msmfdt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001802270>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-msmfdt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000eaef20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-msmfdt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000eaef00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-msmfdt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000eaeee0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-msmfdt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:38:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fd62e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-cwwjhn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b87080>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-cwwjhn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b87060>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-cwwjhn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b87040>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-cwwjhn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fd62e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-cwwjhn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b87080>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-cwwjhn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b87060>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-cwwjhn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b87040>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-cwwjhn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:37:53	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head a484243 · merge 7b9532a	n/a	e2e	failed	#4455 (merged)	9	e2e (9) ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:53 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc0000dfe20>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fbwdrn" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713", "name": "06c30845-f72f-4089-8e52-feb48ef60713", "status": "Failed", "startTime": "2026-03-16T12:56:24.369399603Z", "endTime": "2026-03-16T12:57:38.788738032Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fbwdrn\" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713\",\n \"name\": \"06c30845-f72f-4089-8e52-feb48ef60713\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:24.369399603Z\",\n \"endTime\": \"2026-03-16T12:57:38.788738032Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0000dfe00>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fbwdrn\" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713\",\n \"name\": \"06c30845-f72f-4089-8e52-feb48ef60713\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:24.369399603Z\",\n \"endTime\": \"2026-03-16T12:57:38.788738032Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc0000dfe20>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fbwdrn" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713", "name": "06c30845-f72f-4089-8e52-feb48ef60713", "status": "Failed", "startTime": "2026-03-16T12:56:24.369399603Z", "endTime": "2026-03-16T12:57:38.788738032Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fbwdrn\" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713\",\n \"name\": \"06c30845-f72f-4089-8e52-feb48ef60713\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:24.369399603Z\",\n \"endTime\": \"2026-03-16T12:57:38.788738032Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0000dfe00>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fbwdrn\" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/06c30845-f72f-4089-8e52-feb48ef60713\",\n \"name\": \"06c30845-f72f-4089-8e52-feb48ef60713\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:56:24.369399603Z\",\n \"endTime\": \"2026-03-16T12:57:38.788738032Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-173/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:53 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016241c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-qjzfxw" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000783060>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qjzfxw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000783020>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qjzfxw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000783000>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qjzfxw\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016241c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-qjzfxw" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000783060>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qjzfxw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000783020>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qjzfxw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000783000>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qjzfxw\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:53 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ecc348>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-5bxvxl" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006f7b80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5bxvxl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006f7b60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5bxvxl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006f7b40>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5bxvxl\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ecc348>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-5bxvxl" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006f7b80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5bxvxl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006f7b60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5bxvxl\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006f7b40>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5bxvxl\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:53 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00136c258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-x8tkj7" in resourcegroup="rg-zstream-upgrade-4-20-5zgp48" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ebf1c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-x8tkj7\" in resourcegroup=\"rg-zstream-upgrade-4-20-5zgp48\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ebf1a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-x8tkj7\" in resourcegroup=\"rg-zstream-upgrade-4-20-5zgp48\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ebf180>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-x8tkj7\" in resourcegroup=\"rg-zstream-upgrade-4-20-5zgp48\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00136c258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-x8tkj7" in resourcegroup="rg-zstream-upgrade-4-20-5zgp48" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ebf1c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-x8tkj7\" in resourcegroup=\"rg-zstream-upgrade-4-20-5zgp48\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ebf1a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-x8tkj7\" in resourcegroup=\"rg-zstream-upgrade-4-20-5zgp48\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ebf180>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-x8tkj7\" in resourcegroup=\"rg-zstream-upgrade-4-20-5zgp48\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:53 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d8e228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-6jwdk7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00108b0a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6jwdk7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00108b080>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6jwdk7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00108b060>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6jwdk7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d8e228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-6jwdk7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00108b0a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6jwdk7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00108b080>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6jwdk7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00108b060>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6jwdk7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:53 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000012f30>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-zpzmp2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ea0460>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-zpzmp2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ea0440>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-zpzmp2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ea0420>{ msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-zpzmp2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000012f30>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-zpzmp2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ea0460>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-zpzmp2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ea0440>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-zpzmp2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ea0420>{ msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-zpzmp2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:53 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d34468>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-8jfx2z" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000905a60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8jfx2z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000905a40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8jfx2z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000905a20>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8jfx2z\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d34468>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-8jfx2z" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000905a60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8jfx2z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000905a40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8jfx2z\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000905a20>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-8jfx2z\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred Interrupted by User 11:37:53 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [:0]: Interrupted by User fail [:0]: Interrupted by User ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift 11:37:53 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc000b894c0>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-7wtncf" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07d751a2-253f-4673-8992-6459cd0efabc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07d751a2-253f-4673-8992-6459cd0efabc", "name": "07d751a2-253f-4673-8992-6459cd0efabc", "status": "Failed", "startTime": "2026-03-16T12:32:29.510982837Z", "endTime": "2026-03-16T12:33:20.699146823Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "Failed to get virtual network 'engineering-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7wtncf/providers/Microsoft.Network/virtualNetworks/engineering-vnet-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-7wtncf\" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07d751a2-253f-4673-8992-6459cd0efabc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07d751a2-253f-4673-8992-6459cd0efabc\",\n \"name\": \"07d751a2-253f-4673-8992-6459cd0efabc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:29.510982837Z\",\n \"endTime\": \"2026-03-16T12:33:20.699146823Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'engineering-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7wtncf/providers/Microsoft.Network/virtualNetworks/engineering-vnet-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b894a0>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-7wtncf\" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07d751a2-253f-4673-8992-6459cd0efabc\n-----------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc000b894c0>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-7wtncf" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07d751a2-253f-4673-8992-6459cd0efabc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07d751a2-253f-4673-8992-6459cd0efabc", "name": "07d751a2-253f-4673-8992-6459cd0efabc", "status": "Failed", "startTime": "2026-03-16T12:32:29.510982837Z", "endTime": "2026-03-16T12:33:20.699146823Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "Failed to get virtual network 'engineering-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7wtncf/providers/Microsoft.Network/virtualNetworks/engineering-vnet-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-7wtncf\" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07d751a2-253f-4673-8992-6459cd0efabc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07d751a2-253f-4673-8992-6459cd0efabc\",\n \"name\": \"07d751a2-253f-4673-8992-6459cd0efabc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:29.510982837Z\",\n \"endTime\": \"2026-03-16T12:33:20.699146823Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'engineering-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7wtncf/providers/Microsoft.Network/virtualNetworks/engineering-vnet-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-57/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b894a0>{ msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-7wtncf\" to finish creating: GET https://rp.j6163456.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07d751a2-253f-4673-8992-6459cd0efabc\n-----------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
11:37:46	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c6d34c6	n/a	e2e	failed	★#4457 (open)	16	e2e (16) VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then 11:37:46 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:160]: Unexpected error: <errors.joinError \| 0xc00150e090>: VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then { errs: [ <fmt.wrapError \| 0xc000c42520>{ msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", err: <errors.errorString \| 0xc000834310>{ s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", }, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:160]: Unexpected error: <errors.joinError \| 0xc00150e090>: VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then { errs: [ <fmt.wrapError \| 0xc000c42520>{ msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", err: <errors.errorString \| 0xc000834310>{ s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", }, }, ], } occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014a61e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-rz587x" in resourcegroup="admin-credential-lifecycle-test-52qk2n" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e14e80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-rz587x\" in resourcegroup=\"admin-credential-lifecycle-test-52qk2n\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e14e60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-rz587x\" in resourcegroup=\"admin-credential-lifecycle-test-52qk2n\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e14e40>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-rz587x\" in resourcegroup=\"admin-credential-lifecycle-test-52qk2n\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014a61e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-rz587x" in resourcegroup="admin-credential-lifecycle-test-52qk2n" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e14e80>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-rz587x\" in resourcegroup=\"admin-credential-lifecycle-test-52qk2n\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e14e60>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-rz587x\" in resourcegroup=\"admin-credential-lifecycle-test-52qk2n\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e14e40>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-rz587x\" in resourcegroup=\"admin-credential-lifecycle-test-52qk2n\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True) 11:37:46 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc000f883a8>: context deadline exceeded cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True) { errs: [ <context.deadlineExceededError>{}, <errors.errorString \| 0xc000f6f0f0>{ s: "cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True)", }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc000f883a8>: context deadline exceeded cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True) { errs: [ <context.deadlineExceededError>{}, <errors.errorString \| 0xc000f6f0f0>{ s: "cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True)", }, ], } occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00046a9f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tfvsrr" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000c039e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-tfvsrr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c039c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-tfvsrr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c039a0>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-tfvsrr\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00046a9f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tfvsrr" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000c039e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-tfvsrr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c039c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-tfvsrr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000c039a0>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-tfvsrr\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010da1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-list9967mr" in resourcegroup="ea-list-qnwxh4" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008a73c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list9967mr\" in resourcegroup=\"ea-list-qnwxh4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a73a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list9967mr\" in resourcegroup=\"ea-list-qnwxh4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a7380>{ msg: "failed waiting for hcpCluster=\"ea-list9967mr\" in resourcegroup=\"ea-list-qnwxh4\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010da1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-list9967mr" in resourcegroup="ea-list-qnwxh4" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008a73c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list9967mr\" in resourcegroup=\"ea-list-qnwxh4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a73a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list9967mr\" in resourcegroup=\"ea-list-qnwxh4\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a7380>{ msg: "failed waiting for hcpCluster=\"ea-list9967mr\" in resourcegroup=\"ea-list-qnwxh4\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ebc4e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-j9km84" in resourcegroup="rg-zstream-upgrade-4-21-6dqm5x" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0011eb0e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-j9km84\" in resourcegroup=\"rg-zstream-upgrade-4-21-6dqm5x\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011eb0c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-j9km84\" in resourcegroup=\"rg-zstream-upgrade-4-21-6dqm5x\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011eb0a0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-j9km84\" in resourcegroup=\"rg-zstream-upgrade-4-21-6dqm5x\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ebc4e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-j9km84" in resourcegroup="rg-zstream-upgrade-4-21-6dqm5x" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0011eb0e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-j9km84\" in resourcegroup=\"rg-zstream-upgrade-4-21-6dqm5x\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011eb0c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-j9km84\" in resourcegroup=\"rg-zstream-upgrade-4-21-6dqm5x\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011eb0a0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-j9km84\" in resourcegroup=\"rg-zstream-upgrade-4-21-6dqm5x\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dac330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-wtxg8h" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f0ce40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-wtxg8h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f0ce20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-wtxg8h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f0ce00>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-wtxg8h\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dac330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-wtxg8h" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f0ce40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-wtxg8h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f0ce20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-wtxg8h\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f0ce00>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-wtxg8h\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0011d61b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-qtt68k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000931660>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qtt68k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000931640>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qtt68k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000931620>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qtt68k\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0011d61b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-qtt68k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000931660>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qtt68k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000931640>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qtt68k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000931620>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-qtt68k\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f06138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-zxn7bt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00086b1c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-zxn7bt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00086b1a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-zxn7bt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00086b180>{ msg: "failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-zxn7bt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f06138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-zxn7bt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00086b1c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-zxn7bt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00086b1a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-zxn7bt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00086b180>{ msg: "failed waiting for hcpCluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-zxn7bt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00181c120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-n7mm45" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000fc6dc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-n7mm45\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fc6da0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-n7mm45\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fc6d80>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-n7mm45\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00181c120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-n7mm45" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000fc6dc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-n7mm45\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fc6da0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-n7mm45\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000fc6d80>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-n7mm45\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001007530>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-ccmrgc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a18380>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ccmrgc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a18280>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ccmrgc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a18260>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ccmrgc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001007530>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-ccmrgc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a18380>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ccmrgc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a18280>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ccmrgc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a18260>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-ccmrgc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] 11:37:46 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.001s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Expected <[]string \| len:1, cap:1>: [ "image-registry (Available=False)", ] to be empty fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.001s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Expected <[]string \| len:1, cap:1>: [ "image-registry (Available=False)", ] to be empty failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0008f0b28>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fcpb72" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00096ce60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fcpb72\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00096ce40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fcpb72\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00096ce20>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fcpb72\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0008f0b28>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fcpb72" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00096ce60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fcpb72\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00096ce40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fcpb72\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00096ce20>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fcpb72\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016b8108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-19-bmqbjq" in resourcegroup="rg-zstream-upgrade-4-19-6cgn9b" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00011e100>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-bmqbjq\" in resourcegroup=\"rg-zstream-upgrade-4-19-6cgn9b\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00011e000>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-bmqbjq\" in resourcegroup=\"rg-zstream-upgrade-4-19-6cgn9b\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007bdf00>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-bmqbjq\" in resourcegroup=\"rg-zstream-upgrade-4-19-6cgn9b\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016b8108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-19-bmqbjq" in resourcegroup="rg-zstream-upgrade-4-19-6cgn9b" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00011e100>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-bmqbjq\" in resourcegroup=\"rg-zstream-upgrade-4-19-6cgn9b\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00011e000>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-bmqbjq\" in resourcegroup=\"rg-zstream-upgrade-4-19-6cgn9b\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007bdf00>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-bmqbjq\" in resourcegroup=\"rg-zstream-upgrade-4-19-6cgn9b\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00105a1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-rhfpsw" in resourcegroup="rg-zstream-upgrade-4-20-plc6r2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000acb700>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-rhfpsw\" in resourcegroup=\"rg-zstream-upgrade-4-20-plc6r2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000acb6e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-rhfpsw\" in resourcegroup=\"rg-zstream-upgrade-4-20-plc6r2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000acb6c0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-rhfpsw\" in resourcegroup=\"rg-zstream-upgrade-4-20-plc6r2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00105a1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-rhfpsw" in resourcegroup="rg-zstream-upgrade-4-20-plc6r2" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000acb700>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-rhfpsw\" in resourcegroup=\"rg-zstream-upgrade-4-20-plc6r2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000acb6e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-rhfpsw\" in resourcegroup=\"rg-zstream-upgrade-4-20-plc6r2\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000acb6c0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-rhfpsw\" in resourcegroup=\"rg-zstream-upgrade-4-20-plc6r2\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:46 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010505b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-swh6nb" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008a3840>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-swh6nb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a3820>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-swh6nb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a3800>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-swh6nb\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010505b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-swh6nb" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008a3840>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-swh6nb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a3820>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-swh6nb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008a3800>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-swh6nb\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:37:39	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head b806d77 · merge 9baa329	n/a	e2e	failed	#4461 (merged)	6	e2e (6) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:39 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000782588>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-4m5nhm" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001552de0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-4m5nhm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001552dc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-4m5nhm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001552da0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-4m5nhm\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000782588>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-4m5nhm" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001552de0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-4m5nhm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001552dc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-4m5nhm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001552da0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-4m5nhm\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:39 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:85]: Unexpected error: <fmt.wrapError \| 0xc00145ebe0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-jz875n" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890", "name": "e24d76ce-266a-4022-a92f-2391c52a1890", "status": "Failed", "startTime": "2026-03-16T12:53:41.22993466Z", "endTime": "2026-03-16T12:54:45.622432365Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-jz875n\" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890\",\n \"name\": \"e24d76ce-266a-4022-a92f-2391c52a1890\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:53:41.22993466Z\",\n \"endTime\": \"2026-03-16T12:54:45.622432365Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00145ebc0>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-jz875n\" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890\",\n \"name\": \"e24d76ce-266a-4022-a92f-2391c52a1890\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:53:41.22993466Z\",\n \"endTime\": \"2026-03-16T12:54:45.622432365Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:85]: Unexpected error: <fmt.wrapError \| 0xc00145ebe0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-jz875n" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890", "name": "e24d76ce-266a-4022-a92f-2391c52a1890", "status": "Failed", "startTime": "2026-03-16T12:53:41.22993466Z", "endTime": "2026-03-16T12:54:45.622432365Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-jz875n\" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890\",\n \"name\": \"e24d76ce-266a-4022-a92f-2391c52a1890\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:53:41.22993466Z\",\n \"endTime\": \"2026-03-16T12:54:45.622432365Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00145ebc0>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-jz875n\" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e24d76ce-266a-4022-a92f-2391c52a1890\",\n \"name\": \"e24d76ce-266a-4022-a92f-2391c52a1890\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:53:41.22993466Z\",\n \"endTime\": \"2026-03-16T12:54:45.622432365Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-165/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:39 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00104cdb0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-dlj4bq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00075b180>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-dlj4bq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00075b160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-dlj4bq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00075b140>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-dlj4bq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00104cdb0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-dlj4bq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00075b180>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-dlj4bq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00075b160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-dlj4bq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00075b140>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-dlj4bq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:39 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00012a678>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-qr9pqj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f233c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qr9pqj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f233a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qr9pqj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f23380>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qr9pqj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00012a678>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-qr9pqj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f233c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qr9pqj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f233a0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qr9pqj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f23380>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-qr9pqj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:39 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009d9488>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-6k2ctj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001037060>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-6k2ctj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001037040>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-6k2ctj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001037020>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-6k2ctj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009d9488>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-6k2ctj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001037060>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-6k2ctj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001037040>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-6k2ctj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001037020>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-6k2ctj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:39 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000c25460>: failed to create HCP cluster cluster-zstream-4-20-nr7tz2: failed waiting for cluster="cluster-zstream-4-20-nr7tz2" in resourcegroup="rg-zstream-upgrade-4-20-tn8sgk" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b", "name": "65003440-c6e7-4326-9aee-dee23e620a5b", "status": "Failed", "startTime": "2026-03-16T12:34:40.554390943Z", "endTime": "2026-03-16T12:35:14.145960481Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-nr7tz2: failed waiting for cluster=\"cluster-zstream-4-20-nr7tz2\" in resourcegroup=\"rg-zstream-upgrade-4-20-tn8sgk\" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b\",\n \"name\": \"65003440-c6e7-4326-9aee-dee23e620a5b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:40.554390943Z\",\n \"endTime\": \"2026-03-16T12:35:14.145960481Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c25440>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-nr7tz2\" in resourcegroup=\"rg-zstream-upgrade-4-20-tn8sgk\" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b\",\n \"name\": \"65003440-c6e7-4326-9aee-dee23e620a5b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:40.554390943Z\",\n \"endTime\": \"2026-03-16T12:35:14.145960481Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subsc... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000c25460>: failed to create HCP cluster cluster-zstream-4-20-nr7tz2: failed waiting for cluster="cluster-zstream-4-20-nr7tz2" in resourcegroup="rg-zstream-upgrade-4-20-tn8sgk" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b", "name": "65003440-c6e7-4326-9aee-dee23e620a5b", "status": "Failed", "startTime": "2026-03-16T12:34:40.554390943Z", "endTime": "2026-03-16T12:35:14.145960481Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-nr7tz2: failed waiting for cluster=\"cluster-zstream-4-20-nr7tz2\" in resourcegroup=\"rg-zstream-upgrade-4-20-tn8sgk\" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b\",\n \"name\": \"65003440-c6e7-4326-9aee-dee23e620a5b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:40.554390943Z\",\n \"endTime\": \"2026-03-16T12:35:14.145960481Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c25440>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-nr7tz2\" in resourcegroup=\"rg-zstream-upgrade-4-20-tn8sgk\" to finish creating: GET https://rp.j5970304.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65003440-c6e7-4326-9aee-dee23e620a5b\",\n \"name\": \"65003440-c6e7-4326-9aee-dee23e620a5b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:34:40.554390943Z\",\n \"endTime\": \"2026-03-16T12:35:14.145960481Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-237/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subsc... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
11:37:38	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8bf5305	n/a	e2e	failed	★#4463 (open)	25	e2e (25) ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:124]: Unexpected error: <fmt.wrapError \| 0xc000e46440>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-pullsecret-hcp-cluster-27cvtn" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-pullsecret-hcp-cluster-27cvtn/operationStatuses/08584279348209008846 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-sxgdnx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-p2h32vyw2l2ma\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-pullsecret-hcp-cluster-27cvtn\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-pullsecret-hcp-cluster-27cvtn/operationStatuses/08584279348209008846\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-sxgdnx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-p2h32vyw2l2ma\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e46420>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-pullsecret-hcp-cluster-27cvtn\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-pullsecret-hcp-cluster-27cvtn/operationS... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:124]: Unexpected error: <fmt.wrapError \| 0xc000e46440>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-pullsecret-hcp-cluster-27cvtn" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-pullsecret-hcp-cluster-27cvtn/operationStatuses/08584279348209008846 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-sxgdnx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-p2h32vyw2l2ma\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-pullsecret-hcp-cluster-27cvtn\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-pullsecret-hcp-cluster-27cvtn/operationStatuses/08584279348209008846\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-sxgdnx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-p2h32vyw2l2ma\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e46420>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-pullsecret-hcp-cluster-27cvtn\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-pullsecret-hcp-cluster-27cvtn/operationS... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:77]: Unexpected error: <fmt.wrapError \| 0xc000a61080>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-list-test-cluster-m2fjfs-m7twnq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-list-test-cluster-m2fjfs-m7twnq/operationStatuses/08584279348212172014 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-h64t8x/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3o26pq5ewqal4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-list-test-cluster-m2fjfs-m7twnq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-list-test-cluster-m2fjfs-m7twnq/operationStatuses/08584279348212172014\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-h64t8x/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3o26pq5ewqal4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a61020>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-list-test-cluster-m2fjfs-m7twnq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-list-test-cluster-m2fjfs-m7twnq/op... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:77]: Unexpected error: <fmt.wrapError \| 0xc000a61080>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-list-test-cluster-m2fjfs-m7twnq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-list-test-cluster-m2fjfs-m7twnq/operationStatuses/08584279348212172014 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-h64t8x/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3o26pq5ewqal4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-list-test-cluster-m2fjfs-m7twnq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-list-test-cluster-m2fjfs-m7twnq/operationStatuses/08584279348212172014\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-h64t8x/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3o26pq5ewqal4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a61020>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-list-test-cluster-m2fjfs-m7twnq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-list-test-cluster-m2fjfs-m7twnq/op... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:71]: Unexpected error: <fmt.wrapError \| 0xc000b84a00>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-hcp-cluster-np-128-x9qxl8" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-hcp-cluster-np-128-x9qxl8/operationStatuses/08584279348102075889 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-dv8nvm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ljyjjx2n7gxo\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-hcp-cluster-np-128-x9qxl8\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-hcp-cluster-np-128-x9qxl8/operationStatuses/08584279348102075889\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-dv8nvm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ljyjjx2n7gxo\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can b... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:71]: Unexpected error: <fmt.wrapError \| 0xc000b84a00>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-hcp-cluster-np-128-x9qxl8" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-hcp-cluster-np-128-x9qxl8/operationStatuses/08584279348102075889 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-dv8nvm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ljyjjx2n7gxo\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-hcp-cluster-np-128-x9qxl8\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-hcp-cluster-np-128-x9qxl8/operationStatuses/08584279348102075889\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-dv8nvm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ljyjjx2n7gxo\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can b... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:82]: Unexpected error: <fmt.wrapError \| 0xc000b00720>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-admin-cred-lifecycle-bgf52c-s8crhw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-bgf52c-s8crhw/operationStatuses/08584279348109248089 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-sffqpz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-cxue26wkwsw7m\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-146/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-admin-cred-lifecycle-bgf52c-s8crhw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-bgf52c-s8crhw/operationStatuses/08584279348109248089\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-sffqpz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-cxue26wkwsw7m\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:82]: Unexpected error: <fmt.wrapError \| 0xc000b00720>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-admin-cred-lifecycle-bgf52c-s8crhw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-bgf52c-s8crhw/operationStatuses/08584279348109248089 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-sffqpz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-cxue26wkwsw7m\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-146/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-admin-cred-lifecycle-bgf52c-s8crhw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-bgf52c-s8crhw/operationStatuses/08584279348109248089\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-sffqpz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-cxue26wkwsw7m\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error: <fmt.wrapError \| 0xc00157d1e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-cluster-pkz4xl" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-pkz4xl/operationStatuses/08584279348212078178 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-lqwnfc/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j3gkygvtvkvjs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-cluster-pkz4xl\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-pkz4xl/operationStatuses/08584279348212078178\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-lqwnfc/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j3gkygvtvkvjs\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00157d1c0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-cluster-pkz4xl\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-pkz4xl/operationStatuses/0858427934821207... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error: <fmt.wrapError \| 0xc00157d1e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-cluster-pkz4xl" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-pkz4xl/operationStatuses/08584279348212078178 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-lqwnfc/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j3gkygvtvkvjs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-cluster-pkz4xl\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-pkz4xl/operationStatuses/08584279348212078178\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-lqwnfc/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j3gkygvtvkvjs\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00157d1c0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-cluster-pkz4xl\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-pkz4xl/operationStatuses/0858427934821207... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:117]: Unexpected error: <fmt.wrapError \| 0xc000a48d40>: failed waiting for deployment "mi-cluster-ver-4-19-kq4rvt" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-ver-4-19-kq4rvt/operationStatuses/08584279348212264473 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-mj74kg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-258/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed waiting for deployment \"mi-cluster-ver-4-19-kq4rvt\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-ver-4-19-kq4rvt/operationStatuses/08584279348212264473\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-mj74kg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimi... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:117]: Unexpected error: <fmt.wrapError \| 0xc000a48d40>: failed waiting for deployment "mi-cluster-ver-4-19-kq4rvt" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-ver-4-19-kq4rvt/operationStatuses/08584279348212264473 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-mj74kg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-258/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed waiting for deployment \"mi-cluster-ver-4-19-kq4rvt\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-ver-4-19-kq4rvt/operationStatuses/08584279348212264473\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-mj74kg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimi... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:88]: Unexpected error: <fmt.wrapError \| 0xc000876f00>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-autoscaling-hcp-cluster-4bjpnq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-4bjpnq/operationStatuses/08584279348105316583 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-174/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kqnsz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-autoscaling-hcp-cluster-4bjpnq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-4bjpnq/operationStatuses/08584279348105316583\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-174/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kqnsz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:88]: Unexpected error: <fmt.wrapError \| 0xc000876f00>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-autoscaling-hcp-cluster-4bjpnq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-4bjpnq/operationStatuses/08584279348105316583 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-174/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kqnsz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-autoscaling-hcp-cluster-4bjpnq\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-4bjpnq/operationStatuses/08584279348105316583\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-174/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7kqnsz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:80]: Unexpected error: <fmt.wrapError \| 0xc000e0ebc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-arm64-vm-hcp-cluster-6bz7gn" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-arm64-vm-hcp-cluster-6bz7gn/operationStatuses/08584279348007219168 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-tfhhvz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6htyoibevj7ye\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-arm64-vm-hcp-cluster-6bz7gn\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-arm64-vm-hcp-cluster-6bz7gn/operationStatuses/08584279348007219168\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-tfhhvz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6htyoibevj7ye\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e0eba0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-arm64-vm-hcp-cluster-6bz7gn\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-arm64-vm-hcp-cluster-6bz7gn/operationStatuses... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:80]: Unexpected error: <fmt.wrapError \| 0xc000e0ebc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-arm64-vm-hcp-cluster-6bz7gn" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-arm64-vm-hcp-cluster-6bz7gn/operationStatuses/08584279348007219168 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-tfhhvz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6htyoibevj7ye\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-arm64-vm-hcp-cluster-6bz7gn\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-arm64-vm-hcp-cluster-6bz7gn/operationStatuses/08584279348007219168\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-tfhhvz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6htyoibevj7ye\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e0eba0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-arm64-vm-hcp-cluster-6bz7gn\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-arm64-vm-hcp-cluster-6bz7gn/operationStatuses... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:77]: Unexpected error: <fmt.wrapError \| 0xc000b081c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-oidc-wi-cluster-8dntxt" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-oidc-wi-cluster-8dntxt/operationStatuses/08584279346243357195 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-swjt27/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-s6kwp7xg43dbi\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-11/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-oidc-wi-cluster-8dntxt\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-oidc-wi-cluster-8dntxt/operationStatuses/08584279346243357195\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-swjt27/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-s6kwp7xg43dbi\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/974ebd4... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:77]: Unexpected error: <fmt.wrapError \| 0xc000b081c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-oidc-wi-cluster-8dntxt" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-oidc-wi-cluster-8dntxt/operationStatuses/08584279346243357195 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-swjt27/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-s6kwp7xg43dbi\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-11/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-oidc-wi-cluster-8dntxt\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-oidc-wi-cluster-8dntxt/operationStatuses/08584279346243357195\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-swjt27/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-s6kwp7xg43dbi\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/974ebd4... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:79]: Unexpected error: <fmt.wrapError \| 0xc000b14740>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-8kwprr" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-8kwprr/operationStatuses/08584279346495377975 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-268/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-br4jhg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-8kwprr\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-8kwprr/operationStatuses/08584279346495377975\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-268/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:79]: Unexpected error: <fmt.wrapError \| 0xc000b14740>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-8kwprr" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-8kwprr/operationStatuses/08584279346495377975 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-268/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-br4jhg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-8kwprr\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-8kwprr/operationStatuses/08584279346495377975\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-268/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:75]: Unexpected error: <fmt.wrapError \| 0xc000a804e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-listwq6sgv-xvmrt6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listwq6sgv-xvmrt6/operationStatuses/08584279348213194323 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-klfwp7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3hysavasweej6\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-ea-listwq6sgv-xvmrt6\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listwq6sgv-xvmrt6/operationStatuses/08584279348213194323\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-klfwp7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3hysavasweej6\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a804c0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-ea-listwq6sgv-xvmrt6\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listwq6sgv-xvmrt6/operationStatuses/08584279348213194323\n--------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:75]: Unexpected error: <fmt.wrapError \| 0xc000a804e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-listwq6sgv-xvmrt6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listwq6sgv-xvmrt6/operationStatuses/08584279348213194323 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-klfwp7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3hysavasweej6\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-ea-listwq6sgv-xvmrt6\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listwq6sgv-xvmrt6/operationStatuses/08584279348213194323\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-klfwp7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3hysavasweej6\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a804c0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-ea-listwq6sgv-xvmrt6\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listwq6sgv-xvmrt6/operationStatuses/08584279348213194323\n--------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:77]: Unexpected error: <fmt.wrapError \| 0xc0006afbc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-engineering-hcp-cluster-fwgghx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-engineering-hcp-cluster-fwgghx/operationStatuses/08584279346496343754 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-267/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-engineering-hcp-cluster-fwgghx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-engineering-hcp-cluster-fwgghx/operationStatuses/08584279346496343754\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-267/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0006afba0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-engineering-hcp-cluster-fwgghx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-engineering-hcp-cluster-fwgghx/operationStatuses/08584279346496343754\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-267/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operati... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:77]: Unexpected error: <fmt.wrapError \| 0xc0006afbc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-engineering-hcp-cluster-fwgghx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-engineering-hcp-cluster-fwgghx/operationStatuses/08584279346496343754 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-267/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-engineering-hcp-cluster-fwgghx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-engineering-hcp-cluster-fwgghx/operationStatuses/08584279346496343754\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-267/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0006afba0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-engineering-hcp-cluster-fwgghx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-engineering-hcp-cluster-fwgghx/operationStatuses/08584279346496343754\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-267/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operati... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:89]: Unexpected error: <fmt.wrapError \| 0xc000a123a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-ztbrzp" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-ztbrzp/operationStatuses/08584279348218393707 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9lk62w/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-11/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cidr-connectivity-test-ztbrzp\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-ztbrzp/operationStatuses/08584279348218393707\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9lk62w/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:89]: Unexpected error: <fmt.wrapError \| 0xc000a123a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-ztbrzp" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-ztbrzp/operationStatuses/08584279348218393707 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9lk62w/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-11/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cidr-connectivity-test-ztbrzp\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-ztbrzp/operationStatuses/08584279348218393707\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9lk62w/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:81]: Unexpected error: <fmt.wrapError \| 0xc000b11b20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-sre-hcp-cluster-gblr5n" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-sre-hcp-cluster-gblr5n/operationStatuses/08584279348109332006 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-ngprr7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-456d7mdgworfy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-sre-hcp-cluster-gblr5n\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-sre-hcp-cluster-gblr5n/operationStatuses/08584279348109332006\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-ngprr7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-456d7mdgworfy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b11b00>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-sre-hcp-cluster-gblr5n\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-sre-hcp-cluster-gblr5n/operationStatuses/085842793481093... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:81]: Unexpected error: <fmt.wrapError \| 0xc000b11b20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-sre-hcp-cluster-gblr5n" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-sre-hcp-cluster-gblr5n/operationStatuses/08584279348109332006 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-ngprr7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-456d7mdgworfy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-sre-hcp-cluster-gblr5n\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-sre-hcp-cluster-gblr5n/operationStatuses/08584279348109332006\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-ngprr7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-456d7mdgworfy\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b11b00>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-sre-hcp-cluster-gblr5n\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-sre-hcp-cluster-gblr5n/operationStatuses/085842793481093... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error: <fmt.wrapError \| 0xc000f907c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-sxx96d" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-sxx96d/operationStatuses/08584279348104971436 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-j95xvf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-sxx96d\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-sxx96d/operationStatuses/08584279348104971436\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-j95xvf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f907a0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-sxx96d\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-sxx96d/operationStatuses/0858427934810497... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error: <fmt.wrapError \| 0xc000f907c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-sxx96d" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-sxx96d/operationStatuses/08584279348104971436 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-j95xvf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-sxx96d\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-sxx96d/operationStatuses/08584279348104971436\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-j95xvf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f907a0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-sxx96d\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-sxx96d/operationStatuses/0858427934810497... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000af34c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-dqbcpr-xlfb8n" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-dqbcpr-xlfb8n/operationStatuses/08584279346240555694 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nwdnkm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-opvghki66zxlk\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-dqbcpr-xlfb8n\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-dqbcpr-xlfb8n/operationStatuses/08584279346240555694\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nwdnkm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-opvghki66zxlk\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"N... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000af34c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-dqbcpr-xlfb8n" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-dqbcpr-xlfb8n/operationStatuses/08584279346240555694 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nwdnkm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-opvghki66zxlk\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-dqbcpr-xlfb8n\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-dqbcpr-xlfb8n/operationStatuses/08584279346240555694\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-nwdnkm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-opvghki66zxlk\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"N... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc00083b480>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-256m6w-k2cwhd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-256m6w-k2cwhd/operationStatuses/08584279346244608183 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-6s49sd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-k4eojl4c63cvc\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-146/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-256m6w-k2cwhd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-256m6w-k2cwhd/operationStatuses/08584279346244608183\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-6s49sd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-k4eojl4c63cvc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc00083b480>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-256m6w-k2cwhd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-256m6w-k2cwhd/operationStatuses/08584279346244608183 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-6s49sd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-k4eojl4c63cvc\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-146/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-20-256m6w-k2cwhd\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-256m6w-k2cwhd/operationStatuses/08584279346244608183\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-6s49sd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-k4eojl4c63cvc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error: <fmt.wrapError \| 0xc000820840>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-w8zn5h" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-w8zn5h/operationStatuses/08584279346245680689 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hk2vsv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-174/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-w8zn5h\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-w8zn5h/operationStatuses/08584279346245680689\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hk2vsv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-174/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008207e0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-w8zn5h\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-w8zn5h/operationStatuses/08584279346245680689\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error: <fmt.wrapError \| 0xc000820840>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-w8zn5h" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-w8zn5h/operationStatuses/08584279346245680689 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hk2vsv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-174/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-w8zn5h\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-w8zn5h/operationStatuses/08584279346245680689\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-hk2vsv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-174/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008207e0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-w8zn5h\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-w8zn5h/operationStatuses/08584279346245680689\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: Unexpected error: <fmt.wrapError \| 0xc00079f700>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-9zlgfx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-9zlgfx/operationStatuses/08584279348102707114 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-2bdlsk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-disabled-image-registry-hcp-cluster-9zlgfx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-9zlgfx/operationStatuses/08584279348102707114\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-2bdlsk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"me... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: Unexpected error: <fmt.wrapError \| 0xc00079f700>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-9zlgfx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-9zlgfx/operationStatuses/08584279348102707114 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-2bdlsk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-disabled-image-registry-hcp-cluster-9zlgfx\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-9zlgfx/operationStatuses/08584279348102707114\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-16/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-2bdlsk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"me... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:86]: Unexpected error: <fmt.wrapError \| 0xc0015bada0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-r466zr" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-r466zr/operationStatuses/08584279348107332186 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-q5zbx8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-224/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-r466zr\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-r466zr/operationStatuses/08584279348107332186\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-q5zbx8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:86]: Unexpected error: <fmt.wrapError \| 0xc0015bada0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-r466zr" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-r466zr/operationStatuses/08584279348107332186 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-q5zbx8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-224/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-r466zr\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-r466zr/operationStatuses/08584279348107332186\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-q5zbx8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000ae6600>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-twvswg-cnmbcp" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-twvswg-cnmbcp/operationStatuses/08584279346249234412 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-8dx8gm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-acmzzmpfurenm\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-21-twvswg-cnmbcp\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-twvswg-cnmbcp/operationStatuses/08584279346249234412\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-8dx8gm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-acmzzmpfurenm\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000ae6600>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-twvswg-cnmbcp" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-twvswg-cnmbcp/operationStatuses/08584279346249234412 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-8dx8gm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-acmzzmpfurenm\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-21-twvswg-cnmbcp\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-twvswg-cnmbcp/operationStatuses/08584279346249234412\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-8dx8gm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-acmzzmpfurenm\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc000a82980>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-9cq4wv" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-9cq4wv/operationStatuses/08584279346246022131 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8bkmwp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-277/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-9cq4wv\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-9cq4wv/operationStatuses/08584279346246022131\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8bkmwp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscri... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc000a82980>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-9cq4wv" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-9cq4wv/operationStatuses/08584279346246022131 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8bkmwp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-277/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-9cq4wv\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-9cq4wv/operationStatuses/08584279346246022131\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8bkmwp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscri... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:150]: Unexpected error: <fmt.wrapError \| 0xc001602800>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-9lg58m" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-9lg58m/operationStatuses/08584279348211361650 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-lvsw6z/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-67/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-patch-tags-cluster-9lg58m\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-9lg58m/operationStatuses/08584279348211361650\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-lvsw6z/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscription... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:150]: Unexpected error: <fmt.wrapError \| 0xc001602800>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-9lg58m" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-9lg58m/operationStatuses/08584279348211361650 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-lvsw6z/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-67/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-patch-tags-cluster-9lg58m\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-9lg58m/operationStatuses/08584279348211361650\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-lvsw6z/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscription... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc00069ef40>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-h6rx76" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-h6rx76/operationStatuses/08584279346137168669 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-258/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-nvczjs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-h6rx76\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-h6rx76/operationStatuses/08584279346137168669\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-258/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-nvczjs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:73]: Unexpected error: <fmt.wrapError \| 0xc00069ef40>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-update-nodes-hcp-cluster-h6rx76" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-h6rx76/operationStatuses/08584279346137168669 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-258/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-nvczjs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-update-nodes-hcp-cluster-h6rx76\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-update-nodes-hcp-cluster-h6rx76/operationStatuses/08584279346137168669\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-258/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-nvczjs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-w3k44uxsb3qey\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code RoleAssignmentLimitExceeded; detail message No more role assignments can be created. 11:37:38 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: Unexpected error: <fmt.wrapError \| 0xc000b83260>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-cluster-6tljp8" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-6tljp8/operationStatuses/08584279348216220124 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-6rg4hn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ltdphft7fgrwg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-ea-cluster-6tljp8\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-6tljp8/operationStatuses/08584279348216220124\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-6rg4hn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ltdphft7fgrwg\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b83240>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-ea-cluster-6tljp8\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-6tljp8/operationStatuses/08584279348216220124\n------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: Unexpected error: <fmt.wrapError \| 0xc000b83260>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-cluster-6tljp8" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-6tljp8/operationStatuses/08584279348216220124 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-6rg4hn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ltdphft7fgrwg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n },\r\n {\r\n \"code\": \"RoleAssignmentLimitExceeded\",\r\n \"message\": \"No more role assignments can be created.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-ea-cluster-6tljp8\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-6tljp8/operationStatuses/08584279348216220124\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-6rg4hn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ltdphft7fgrwg\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"RoleAssignmentLimitExceeded\\\",\\r\\n \\\"message\\\": \\\"No more role assignments can be created.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b83240>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-ea-cluster-6tljp8\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-6tljp8/operationStatuses/08584279348216220124\n------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
11:37:31	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 842e66a	n/a	e2e	failed	★#4465 (closed)	14	e2e (14) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e2a210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m7l49f" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a1a8e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7l49f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1a840>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7l49f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1a780>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7l49f\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e2a210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m7l49f" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a1a8e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7l49f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1a840>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7l49f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1a780>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-m7l49f\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:31 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc000440320>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-bc8srv" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd", "name": "cdc41c88-18b6-498f-8f84-982adcd4f3fd", "status": "Failed", "startTime": "2026-03-16T12:41:45.16428613Z", "endTime": "2026-03-16T12:42:47.486090571Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-bc8srv\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd\",\n \"name\": \"cdc41c88-18b6-498f-8f84-982adcd4f3fd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:41:45.16428613Z\",\n \"endTime\": \"2026-03-16T12:42:47.486090571Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0004402c0>{ msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-bc8srv\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd\",\n \"name\": \"cdc41c88-18b6-498f-8f84-982adcd4f3fd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:41:45.16428613Z\",\n \"endTime\": \"2026-03-16T12:42:47.486090571Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc000440320>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-bc8srv" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd", "name": "cdc41c88-18b6-498f-8f84-982adcd4f3fd", "status": "Failed", "startTime": "2026-03-16T12:41:45.16428613Z", "endTime": "2026-03-16T12:42:47.486090571Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-bc8srv\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd\",\n \"name\": \"cdc41c88-18b6-498f-8f84-982adcd4f3fd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:41:45.16428613Z\",\n \"endTime\": \"2026-03-16T12:42:47.486090571Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0004402c0>{ msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-bc8srv\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cdc41c88-18b6-498f-8f84-982adcd4f3fd\",\n \"name\": \"cdc41c88-18b6-498f-8f84-982adcd4f3fd\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:41:45.16428613Z\",\n \"endTime\": \"2026-03-16T12:42:47.486090571Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:31 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0007bc4c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-qmb28v" in resourcegroup="rg-zstream-upgrade-4-20-2v76gp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006f7ca0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-qmb28v\" in resourcegroup=\"rg-zstream-upgrade-4-20-2v76gp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006f7c80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-qmb28v\" in resourcegroup=\"rg-zstream-upgrade-4-20-2v76gp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006f7900>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-qmb28v\" in resourcegroup=\"rg-zstream-upgrade-4-20-2v76gp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0007bc4c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-qmb28v" in resourcegroup="rg-zstream-upgrade-4-20-2v76gp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006f7ca0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-qmb28v\" in resourcegroup=\"rg-zstream-upgrade-4-20-2v76gp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006f7c80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-qmb28v\" in resourcegroup=\"rg-zstream-upgrade-4-20-2v76gp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006f7900>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-qmb28v\" in resourcegroup=\"rg-zstream-upgrade-4-20-2v76gp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:31 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc0008b7280>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-7rjp56" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9", "name": "164046a3-fff4-4265-b9e3-481353a864f9", "status": "Failed", "startTime": "2026-03-16T12:46:46.289122417Z", "endTime": "2026-03-16T12:47:33.920097986Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-7rjp56\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9\",\n \"name\": \"164046a3-fff4-4265-b9e3-481353a864f9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:46:46.289122417Z\",\n \"endTime\": \"2026-03-16T12:47:33.920097986Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008b7260>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-7rjp56\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9\",\n \"name\": \"164046a3-fff4-4265-b9e3-481353a864f9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:46:46.289122417Z\",\n \"endTime\": \"2026-03-16T12:47:33.920097986Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc0008b7280>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-7rjp56" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9", "name": "164046a3-fff4-4265-b9e3-481353a864f9", "status": "Failed", "startTime": "2026-03-16T12:46:46.289122417Z", "endTime": "2026-03-16T12:47:33.920097986Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-7rjp56\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9\",\n \"name\": \"164046a3-fff4-4265-b9e3-481353a864f9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:46:46.289122417Z\",\n \"endTime\": \"2026-03-16T12:47:33.920097986Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008b7260>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-7rjp56\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/164046a3-fff4-4265-b9e3-481353a864f9\",\n \"name\": \"164046a3-fff4-4265-b9e3-481353a864f9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:46:46.289122417Z\",\n \"endTime\": \"2026-03-16T12:47:33.920097986Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e1c1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2s8ksj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e27060>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2s8ksj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e27040>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2s8ksj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e27020>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2s8ksj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e1c1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2s8ksj" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e27060>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2s8ksj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e27040>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2s8ksj\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e27020>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2s8ksj\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True) 11:37:31 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc001101680>: context deadline exceeded cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True) { errs: [ <context.deadlineExceededError>{}, <errors.errorString \| 0xc0006ec9b0>{ s: "cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True)", }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc001101680>: context deadline exceeded cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True) { errs: [ <context.deadlineExceededError>{}, <errors.errorString \| 0xc0006ec9b0>{ s: "cluster operators not available: image-registry (Available=False, Progressing=True, Degraded=True)", }, ], } occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapError \| 0xc00075d000>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-49g9c7" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f7e5df7-d77c-4719-ab29-57d47b6ec43e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f7e5df7-d77c-4719-ab29-57d47b6ec43e", "name": "5f7e5df7-d77c-4719-ab29-57d47b6ec43e", "status": "Failed", "startTime": "2026-03-16T12:32:46.202033733Z", "endTime": "2026-03-16T12:33:12.711755824Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-49g9c7\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f7e5df7-d77c-4719-ab29-57d47b6ec43e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f7e5df7-d77c-4719-ab29-57d47b6ec43e\",\n \"name\": \"5f7e5df7-d77c-4719-ab29-57d47b6ec43e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:46.202033733Z\",\n \"endTime\": \"2026-03-16T12:33:12.711755824Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001488510>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 12:33:16 GMT", ], "Content-Length": ["1561"], "X-Envoy-Upstream-Service-Time": ["57"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "e5d2fd0e-17e5-49ac-ac1c-60489cae0e3f", ], }, Body: <exported.nopClosingBytesReader \| 0xc00075c9c0>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 53, 102, 55, 101, 53, 100, 102, 55, 45, 100, 55, 55, 99, 45, 52, 55, 49, 57, 45, 97, 98, 50, 57, 45, 53, 55, 100, 52, 55, 98, 54, 101, 99, 52, 51, 101, 34, 44, 10, 32, 32, 32, 32, 34, 11... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapError \| 0xc00075d000>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-49g9c7" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f7e5df7-d77c-4719-ab29-57d47b6ec43e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f7e5df7-d77c-4719-ab29-57d47b6ec43e", "name": "5f7e5df7-d77c-4719-ab29-57d47b6ec43e", "status": "Failed", "startTime": "2026-03-16T12:32:46.202033733Z", "endTime": "2026-03-16T12:33:12.711755824Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-49g9c7\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f7e5df7-d77c-4719-ab29-57d47b6ec43e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5f7e5df7-d77c-4719-ab29-57d47b6ec43e\",\n \"name\": \"5f7e5df7-d77c-4719-ab29-57d47b6ec43e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:46.202033733Z\",\n \"endTime\": \"2026-03-16T12:33:12.711755824Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001488510>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 12:33:16 GMT", ], "Content-Length": ["1561"], "X-Envoy-Upstream-Service-Time": ["57"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "e5d2fd0e-17e5-49ac-ac1c-60489cae0e3f", ], }, Body: <exported.nopClosingBytesReader \| 0xc00075c9c0>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 53, 102, 55, 101, 53, 100, 102, 55, 45, 100, 55, 55, 99, 45, 52, 55, 49, 57, 45, 97, 98, 50, 57, 45, 53, 55, 100, 52, 55, 98, 54, 101, 99, 52, 51, 101, 34, 44, 10, 32, 32, 32, 32, 34, 11... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:31 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00105e300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-ffm6zs" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001692f40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ffm6zs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001692f20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ffm6zs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001692f00>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ffm6zs\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00105e300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-ffm6zs" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001692f40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ffm6zs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001692f20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ffm6zs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001692f00>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ffm6zs\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:31 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009de168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-8cnmbs" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008896e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-8cnmbs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008896c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-8cnmbs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008896a0>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-8cnmbs\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009de168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-8cnmbs" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008896e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-8cnmbs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008896c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-8cnmbs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008896a0>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-8cnmbs\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:31 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc000492960>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-6lzq82" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2", "name": "559203f1-1420-4ab3-b051-c70ea7bf34e2", "status": "Failed", "startTime": "2026-03-16T12:36:11.314175626Z", "endTime": "2026-03-16T12:37:05.885517874Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-6lzq82\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2\",\n \"name\": \"559203f1-1420-4ab3-b051-c70ea7bf34e2\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:11.314175626Z\",\n \"endTime\": \"2026-03-16T12:37:05.885517874Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000492920>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-6lzq82\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2\",\n \"name\": \"559203f1-1420-4ab3-b051-c70ea7bf34e2\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:11.314175626Z\",\n \"endTime\": \"2026-03-16T12:37:05.885517874Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc000492960>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-6lzq82" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2", "name": "559203f1-1420-4ab3-b051-c70ea7bf34e2", "status": "Failed", "startTime": "2026-03-16T12:36:11.314175626Z", "endTime": "2026-03-16T12:37:05.885517874Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-6lzq82\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2\",\n \"name\": \"559203f1-1420-4ab3-b051-c70ea7bf34e2\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:11.314175626Z\",\n \"endTime\": \"2026-03-16T12:37:05.885517874Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000492920>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-6lzq82\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/559203f1-1420-4ab3-b051-c70ea7bf34e2\",\n \"name\": \"559203f1-1420-4ab3-b051-c70ea7bf34e2\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:11.314175626Z\",\n \"endTime\": \"2026-03-16T12:37:05.885517874Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-0/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:31 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc00074a480>: failed to create HCP cluster cluster-zstream-4-21-p8thq2: failed waiting for cluster="cluster-zstream-4-21-p8thq2" in resourcegroup="rg-zstream-upgrade-4-21-kpc4hw" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703", "name": "8e1cf903-ac74-4008-9844-616ceb275703", "status": "Failed", "startTime": "2026-03-16T12:36:47.414774401Z", "endTime": "2026-03-16T12:38:15.590235552Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-p8thq2: failed waiting for cluster=\"cluster-zstream-4-21-p8thq2\" in resourcegroup=\"rg-zstream-upgrade-4-21-kpc4hw\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703\",\n \"name\": \"8e1cf903-ac74-4008-9844-616ceb275703\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:47.414774401Z\",\n \"endTime\": \"2026-03-16T12:38:15.590235552Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00074a460>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-p8thq2\" in resourcegroup=\"rg-zstream-upgrade-4-21-kpc4hw\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703\",\n \"name\": \"8e1cf903-ac74-4008-9844-616ceb275703\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:47.414774401Z\",\n \"endTime\": \"2026-03-16T12:38:15.590235552Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subscript... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc00074a480>: failed to create HCP cluster cluster-zstream-4-21-p8thq2: failed waiting for cluster="cluster-zstream-4-21-p8thq2" in resourcegroup="rg-zstream-upgrade-4-21-kpc4hw" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703", "name": "8e1cf903-ac74-4008-9844-616ceb275703", "status": "Failed", "startTime": "2026-03-16T12:36:47.414774401Z", "endTime": "2026-03-16T12:38:15.590235552Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-p8thq2: failed waiting for cluster=\"cluster-zstream-4-21-p8thq2\" in resourcegroup=\"rg-zstream-upgrade-4-21-kpc4hw\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703\",\n \"name\": \"8e1cf903-ac74-4008-9844-616ceb275703\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:47.414774401Z\",\n \"endTime\": \"2026-03-16T12:38:15.590235552Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00074a460>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-p8thq2\" in resourcegroup=\"rg-zstream-upgrade-4-21-kpc4hw\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8e1cf903-ac74-4008-9844-616ceb275703\",\n \"name\": \"8e1cf903-ac74-4008-9844-616ceb275703\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:36:47.414774401Z\",\n \"endTime\": \"2026-03-16T12:38:15.590235552Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-81/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subscript... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:31 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ed04b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listtjr4ps" in resourcegroup="ea-list-6vpjv7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00076bdc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listtjr4ps\" in resourcegroup=\"ea-list-6vpjv7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00076bda0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listtjr4ps\" in resourcegroup=\"ea-list-6vpjv7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00076bd80>{ msg: "failed waiting for hcpCluster=\"ea-listtjr4ps\" in resourcegroup=\"ea-list-6vpjv7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ed04b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listtjr4ps" in resourcegroup="ea-list-6vpjv7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00076bdc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listtjr4ps\" in resourcegroup=\"ea-list-6vpjv7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00076bda0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listtjr4ps\" in resourcegroup=\"ea-list-6vpjv7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00076bd80>{ msg: "failed waiting for hcpCluster=\"ea-listtjr4ps\" in resourcegroup=\"ea-list-6vpjv7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:31 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error: <fmt.wrapError \| 0xc000821180>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-q8n88z" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b", "name": "73331c6d-bdfd-4ef6-85d5-b0272f79589b", "status": "Failed", "startTime": "2026-03-16T12:32:45.831342796Z", "endTime": "2026-03-16T12:33:12.654919141Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-q8n88z\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b\",\n \"name\": \"73331c6d-bdfd-4ef6-85d5-b0272f79589b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:45.831342796Z\",\n \"endTime\": \"2026-03-16T12:33:12.654919141Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000821160>{ msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-q8n88z\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b\",\n \"name\": \"73331c6d-bdfd-4ef6-85d5-b0272f79589b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:45.831342796Z\",\n \"endTime\": \"2026-03-16T12:33:12.654919141Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error: <fmt.wrapError \| 0xc000821180>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-q8n88z" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b", "name": "73331c6d-bdfd-4ef6-85d5-b0272f79589b", "status": "Failed", "startTime": "2026-03-16T12:32:45.831342796Z", "endTime": "2026-03-16T12:33:12.654919141Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-q8n88z\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b\",\n \"name\": \"73331c6d-bdfd-4ef6-85d5-b0272f79589b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:45.831342796Z\",\n \"endTime\": \"2026-03-16T12:33:12.654919141Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000821160>{ msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-q8n88z\" to finish creating: GET https://rp.j7919616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73331c6d-bdfd-4ef6-85d5-b0272f79589b\",\n \"name\": \"73331c6d-bdfd-4ef6-85d5-b0272f79589b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:32:45.831342796Z\",\n \"endTime\": \"2026-03-16T12:33:12.654919141Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-8/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d86f90>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nn74pg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b0fa00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nn74pg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b0f980>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nn74pg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b0f960>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nn74pg\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d86f90>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nn74pg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b0fa00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nn74pg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b0f980>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nn74pg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b0f960>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nn74pg\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:37:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 710411a · merge 8cb4bd5	n/a	e2e	failed	#4466 (merged)	10	e2e (10) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:23 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016d81e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-sr7fvl" in resourcegroup="rg-zstream-upgrade-4-20-tfrc9m" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f90fe0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-sr7fvl\" in resourcegroup=\"rg-zstream-upgrade-4-20-tfrc9m\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f90fc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-sr7fvl\" in resourcegroup=\"rg-zstream-upgrade-4-20-tfrc9m\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f90fa0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-sr7fvl\" in resourcegroup=\"rg-zstream-upgrade-4-20-tfrc9m\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016d81e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-sr7fvl" in resourcegroup="rg-zstream-upgrade-4-20-tfrc9m" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f90fe0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-sr7fvl\" in resourcegroup=\"rg-zstream-upgrade-4-20-tfrc9m\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f90fc0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-20-sr7fvl\" in resourcegroup=\"rg-zstream-upgrade-4-20-tfrc9m\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f90fa0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-sr7fvl\" in resourcegroup=\"rg-zstream-upgrade-4-20-tfrc9m\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:23 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010f8348>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-k8zshn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008d3ba0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k8zshn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008d3b80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k8zshn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008d3b60>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k8zshn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010f8348>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-k8zshn" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008d3ba0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k8zshn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008d3b80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k8zshn\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008d3b60>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k8zshn\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:23 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0013c6180>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-x864pq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000135440>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-x864pq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000135420>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-x864pq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0001353e0>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-x864pq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0013c6180>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-x864pq" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000135440>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-x864pq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000135420>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-x864pq\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0001353e0>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-x864pq\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:23 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc000e6c4c0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-8svmbp" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88", "name": "73dcdb23-9553-4247-929b-fe1cac143a88", "status": "Failed", "startTime": "2026-03-16T12:49:19.684427225Z", "endTime": "2026-03-16T12:50:43.423575122Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-8svmbp\" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88\",\n \"name\": \"73dcdb23-9553-4247-929b-fe1cac143a88\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:19.684427225Z\",\n \"endTime\": \"2026-03-16T12:50:43.423575122Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e6c4a0>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-8svmbp\" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88\",\n \"name\": \"73dcdb23-9553-4247-929b-fe1cac143a88\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:19.684427225Z\",\n \"endTime\": \"2026-03-16T12:50:43.423575122Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc000e6c4c0>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-8svmbp" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88", "name": "73dcdb23-9553-4247-929b-fe1cac143a88", "status": "Failed", "startTime": "2026-03-16T12:49:19.684427225Z", "endTime": "2026-03-16T12:50:43.423575122Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-8svmbp\" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88\",\n \"name\": \"73dcdb23-9553-4247-929b-fe1cac143a88\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:19.684427225Z\",\n \"endTime\": \"2026-03-16T12:50:43.423575122Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e6c4a0>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-8svmbp\" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/73dcdb23-9553-4247-929b-fe1cac143a88\",\n \"name\": \"73dcdb23-9553-4247-929b-fe1cac143a88\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:19.684427225Z\",\n \"endTime\": \"2026-03-16T12:50:43.423575122Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-83/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0011ee2e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-jwm7pz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009a9bc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jwm7pz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009a9ba0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jwm7pz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009a9b00>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jwm7pz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0011ee2e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-jwm7pz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0009a9bc0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jwm7pz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009a9ba0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jwm7pz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0009a9b00>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jwm7pz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:23 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapError \| 0xc001340320>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qd594h" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256", "name": "7f8e94da-9c36-4738-8107-1faf2b611256", "status": "Failed", "startTime": "2026-03-16T12:26:39.838085664Z", "endTime": "2026-03-16T12:27:14.55398244Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-qd594h\" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256\",\n \"name\": \"7f8e94da-9c36-4738-8107-1faf2b611256\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:26:39.838085664Z\",\n \"endTime\": \"2026-03-16T12:27:14.55398244Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001340300>{ msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-qd594h\" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256\",\n \"name\": \"7f8e94da-9c36-4738-8107-1faf2b611256\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:26:39.838085664Z\",\n \"endTime\": \"2026-03-16T12:27:14.55398244Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapError \| 0xc001340320>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-qd594h" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256", "name": "7f8e94da-9c36-4738-8107-1faf2b611256", "status": "Failed", "startTime": "2026-03-16T12:26:39.838085664Z", "endTime": "2026-03-16T12:27:14.55398244Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-qd594h\" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256\",\n \"name\": \"7f8e94da-9c36-4738-8107-1faf2b611256\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:26:39.838085664Z\",\n \"endTime\": \"2026-03-16T12:27:14.55398244Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001340300>{ msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-qd594h\" to finish creating: GET https://rp.j4390272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7f8e94da-9c36-4738-8107-1faf2b611256\",\n \"name\": \"7f8e94da-9c36-4738-8107-1faf2b611256\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:26:39.838085664Z\",\n \"endTime\": \"2026-03-16T12:27:14.55398244Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-228/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:23 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001070330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fwdxl9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00067a840>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fwdxl9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00067a820>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fwdxl9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00067a800>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fwdxl9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001070330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fwdxl9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00067a840>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fwdxl9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00067a820>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fwdxl9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00067a800>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-fwdxl9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:23 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009aa330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-kszlkh" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00167af40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kszlkh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00167af20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kszlkh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00167af00>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kszlkh\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009aa330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-kszlkh" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00167af40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kszlkh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00167af20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kszlkh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00167af00>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kszlkh\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:23 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00017a8a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-r6hhrs" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00013d580>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-r6hhrs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00013d560>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-r6hhrs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00013d540>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-r6hhrs\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00017a8a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-r6hhrs" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00013d580>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-r6hhrs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00013d560>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-r6hhrs\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00013d540>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-r6hhrs\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:23 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0015e40a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-92hzhp" in resourcegroup="rg-zstream-upgrade-4-21-6847r5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e94e00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-92hzhp\" in resourcegroup=\"rg-zstream-upgrade-4-21-6847r5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e94de0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-92hzhp\" in resourcegroup=\"rg-zstream-upgrade-4-21-6847r5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e94dc0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-92hzhp\" in resourcegroup=\"rg-zstream-upgrade-4-21-6847r5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0015e40a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-92hzhp" in resourcegroup="rg-zstream-upgrade-4-21-6847r5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e94e00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-92hzhp\" in resourcegroup=\"rg-zstream-upgrade-4-21-6847r5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e94de0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-92hzhp\" in resourcegroup=\"rg-zstream-upgrade-4-21-6847r5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e94dc0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-92hzhp\" in resourcegroup=\"rg-zstream-upgrade-4-21-6847r5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:37:16	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4834b3b	n/a	e2e	failed	★#4467 (closed)	15	e2e (15) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:16 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000edce58>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-cs5465" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008c77a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-cs5465\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008c7780>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-cs5465\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008c7760>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-cs5465\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000edce58>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-cs5465" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008c77a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-cs5465\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008c7780>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-cs5465\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008c7760>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-cs5465\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed 11:37:16 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:152]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:152]: Cluster provisioning failed ERROR CODE: InternalServerError; detail code 429; detail message Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds.; provider Microsoft.ManagedIdentity 11:37:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapError \| 0xc000d9c5c0>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-bh8k9w" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/507fce4f-c4c5-40f4-a7c1-39b34b540b30 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/507fce4f-c4c5-40f4-a7c1-39b34b540b30", "name": "507fce4f-c4c5-40f4-a7c1-39b34b540b30", "status": "Failed", "startTime": "2026-03-16T14:29:19.556311137Z", "endTime": "2026-03-16T14:30:55.368968887Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-bh8k9w\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/507fce4f-c4c5-40f4-a7c1-39b34b540b30\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/507fce4f-c4c5-40f4-a7c1-39b34b540b30\",\n \"name\": \"507fce4f-c4c5-40f4-a7c1-39b34b540b30\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:19.556311137Z\",\n \"endTime\": \"2026-03-16T14:30:55.368968887Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001488870>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["10"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "070079fb-42c9-4860-a33d-946a989c0339", ], "Date": [ "Mon, 16 Mar 2026 14:31:00 GMT", ], "Content-Length": ["1396"], }, Body: <exported.nopClosingBytesReader \| 0xc000d9c3a0>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 53, 48, 55, 102, 99, 101, 52, 102, 45, 99, 52, 99, 53, 45, 52, 48, 102, 52, 45, 97, 55, 99, 49, 45, 51, 57, 98, 51, 52, 98, 53, 52, 48, 98, 51, 48, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 53, 48, 55, 102, 99, 101, 52, 102, 45, 99, 52, 99, 53, 45, 52, 48, 102, 52, 45, 97, 55, 99, 49, 45, 51, 57, 98, 51, 52, 98, 53, 52, 48, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapError \| 0xc000d9c5c0>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-bh8k9w" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/507fce4f-c4c5-40f4-a7c1-39b34b540b30 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/507fce4f-c4c5-40f4-a7c1-39b34b540b30", "name": "507fce4f-c4c5-40f4-a7c1-39b34b540b30", "status": "Failed", "startTime": "2026-03-16T14:29:19.556311137Z", "endTime": "2026-03-16T14:30:55.368968887Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-bh8k9w\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/507fce4f-c4c5-40f4-a7c1-39b34b540b30\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/507fce4f-c4c5-40f4-a7c1-39b34b540b30\",\n \"name\": \"507fce4f-c4c5-40f4-a7c1-39b34b540b30\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:19.556311137Z\",\n \"endTime\": \"2026-03-16T14:30:55.368968887Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001488870>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["10"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "070079fb-42c9-4860-a33d-946a989c0339", ], "Date": [ "Mon, 16 Mar 2026 14:31:00 GMT", ], "Content-Length": ["1396"], }, Body: <exported.nopClosingBytesReader \| 0xc000d9c3a0>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 53, 48, 55, 102, 99, 101, 52, 102, 45, 99, 52, 99, 53, 45, 52, 48, 102, 52, 45, 97, 55, 99, 49, 45, 51, 57, 98, 51, 52, 98, 53, 52, 48, 98, 51, 48, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 53, 48, 55, 102, 99, 101, 52, 102, 45, 99, 52, 99, 53, 45, 52, 48, 102, 52, 45, 97, 55, 99, 49, 45, 51, 57, 98, 51, 52, 98, 53, 52, 48, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code 429; detail message Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds.; provider Microsoft.ManagedIdentity 11:37:16 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc0009f5520>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-r4548l" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968", "name": "7821c08d-3139-459a-921e-3f15f32a1968", "status": "Failed", "startTime": "2026-03-16T14:29:29.983068909Z", "endTime": "2026-03-16T14:30:55.403589972Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-r4548l\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968\",\n \"name\": \"7821c08d-3139-459a-921e-3f15f32a1968\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:29.983068909Z\",\n \"endTime\": \"2026-03-16T14:30:55.403589972Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0009f5500>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-r4548l\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968\",\n \"name\": \"7821c08d-3139-459a-921e-3f15f32a1968\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:29.983068909Z\",\n \"endTime\": \"2026-03-16T14:30:55.403589972Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued t... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc0009f5520>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-r4548l" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968", "name": "7821c08d-3139-459a-921e-3f15f32a1968", "status": "Failed", "startTime": "2026-03-16T14:29:29.983068909Z", "endTime": "2026-03-16T14:30:55.403589972Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-r4548l\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968\",\n \"name\": \"7821c08d-3139-459a-921e-3f15f32a1968\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:29.983068909Z\",\n \"endTime\": \"2026-03-16T14:30:55.403589972Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 2 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0009f5500>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-r4548l\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7821c08d-3139-459a-921e-3f15f32a1968\",\n \"name\": \"7821c08d-3139-459a-921e-3f15f32a1968\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:29.983068909Z\",\n \"endTime\": \"2026-03-16T14:30:55.403589972Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-155/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued t... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016aa300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-glhwtw" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a21880>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-glhwtw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a21860>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-glhwtw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a21840>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-glhwtw\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016aa300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-glhwtw" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a21880>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-glhwtw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a21860>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-glhwtw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a21840>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-glhwtw\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00127c060>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-7bhc2w" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006c5ae0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-7bhc2w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006c5aa0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-7bhc2w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006c5a80>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-7bhc2w\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00127c060>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-7bhc2w" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0006c5ae0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-7bhc2w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006c5aa0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-7bhc2w\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006c5a80>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-7bhc2w\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code 429; detail message Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds.; provider Microsoft.ManagedIdentity 11:37:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapError \| 0xc000c10640>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-6qd28n" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59", "name": "ddb4a419-8db1-4ddb-8221-c56bc2816e59", "status": "Failed", "startTime": "2026-03-16T14:29:32.981550365Z", "endTime": "2026-03-16T14:30:55.380145231Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-6qd28n\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59\",\n \"name\": \"ddb4a419-8db1-4ddb-8221-c56bc2816e59\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:32.981550365Z\",\n \"endTime\": \"2026-03-16T14:30:55.380145231Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c10620>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-6qd28n\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59\",\n \"name\": \"ddb4a419-8db1-4ddb-8221-c56bc2816e59\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:32.981550365Z\",\n \"endTime\": \"2026-03-16T14:30:55.380145231Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription 974ebd46-8ad3-41e3-afef-7... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapError \| 0xc000c10640>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-6qd28n" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59", "name": "ddb4a419-8db1-4ddb-8221-c56bc2816e59", "status": "Failed", "startTime": "2026-03-16T14:29:32.981550365Z", "endTime": "2026-03-16T14:30:55.380145231Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: 429\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"429\",\n \"message\": \"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-6qd28n\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59\",\n \"name\": \"ddb4a419-8db1-4ddb-8221-c56bc2816e59\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:32.981550365Z\",\n \"endTime\": \"2026-03-16T14:30:55.380145231Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c10620>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-6qd28n\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddb4a419-8db1-4ddb-8221-c56bc2816e59\",\n \"name\": \"ddb4a419-8db1-4ddb-8221-c56bc2816e59\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:29:32.981550365Z\",\n \"endTime\": \"2026-03-16T14:30:55.380145231Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-161/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: 429\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"429\\\",\\n \\\"message\\\": \\\"Request is temporarily throttled because subscription 974ebd46-8ad3-41e3-afef-7... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:16 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016ac138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-w8fwbm" in resourcegroup="rg-zstream-upgrade-4-21-4p9fwg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e0b7e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-w8fwbm\" in resourcegroup=\"rg-zstream-upgrade-4-21-4p9fwg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e0b7c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-w8fwbm\" in resourcegroup=\"rg-zstream-upgrade-4-21-4p9fwg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e0b7a0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-w8fwbm\" in resourcegroup=\"rg-zstream-upgrade-4-21-4p9fwg\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0016ac138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-w8fwbm" in resourcegroup="rg-zstream-upgrade-4-21-4p9fwg" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000e0b7e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-w8fwbm\" in resourcegroup=\"rg-zstream-upgrade-4-21-4p9fwg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e0b7c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-21-w8fwbm\" in resourcegroup=\"rg-zstream-upgrade-4-21-4p9fwg\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e0b7a0>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-w8fwbm\" in resourcegroup=\"rg-zstream-upgrade-4-21-4p9fwg\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:16 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc001434a80>: failed to create HCP cluster cluster-zstream-4-20-vxx48m: failed waiting for cluster="cluster-zstream-4-20-vxx48m" in resourcegroup="rg-zstream-upgrade-4-20-px5chb" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746", "name": "4e8e1de0-accf-4251-9554-eef8a21cb746", "status": "Failed", "startTime": "2026-03-16T14:34:51.663043024Z", "endTime": "2026-03-16T14:36:01.013234838Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-vxx48m: failed waiting for cluster=\"cluster-zstream-4-20-vxx48m\" in resourcegroup=\"rg-zstream-upgrade-4-20-px5chb\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746\",\n \"name\": \"4e8e1de0-accf-4251-9554-eef8a21cb746\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:34:51.663043024Z\",\n \"endTime\": \"2026-03-16T14:36:01.013234838Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001434a60>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-vxx48m\" in resourcegroup=\"rg-zstream-upgrade-4-20-px5chb\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746\",\n \"name\": \"4e8e1de0-accf-4251-9554-eef8a21cb746\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:34:51.663043024Z\",\n \"endTime\": \"2026-03-16T14:36:01.013234838Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc001434a80>: failed to create HCP cluster cluster-zstream-4-20-vxx48m: failed waiting for cluster="cluster-zstream-4-20-vxx48m" in resourcegroup="rg-zstream-upgrade-4-20-px5chb" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746", "name": "4e8e1de0-accf-4251-9554-eef8a21cb746", "status": "Failed", "startTime": "2026-03-16T14:34:51.663043024Z", "endTime": "2026-03-16T14:36:01.013234838Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-vxx48m: failed waiting for cluster=\"cluster-zstream-4-20-vxx48m\" in resourcegroup=\"rg-zstream-upgrade-4-20-px5chb\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746\",\n \"name\": \"4e8e1de0-accf-4251-9554-eef8a21cb746\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:34:51.663043024Z\",\n \"endTime\": \"2026-03-16T14:36:01.013234838Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001434a60>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-vxx48m\" in resourcegroup=\"rg-zstream-upgrade-4-20-px5chb\" to finish creating: GET https://rp.j7726720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e8e1de0-accf-4251-9554-eef8a21cb746\",\n \"name\": \"4e8e1de0-accf-4251-9554-eef8a21cb746\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T14:34:51.663043024Z\",\n \"endTime\": \"2026-03-16T14:36:01.013234838Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-255/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:16 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e4c288>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-jlbmd5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0016b4ea0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-jlbmd5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0016b4e80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-jlbmd5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0016b4e60>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-jlbmd5\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000e4c288>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-jlbmd5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0016b4ea0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-jlbmd5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0016b4e80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-jlbmd5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0016b4e60>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-jlbmd5\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] 11:37:16 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.004s. Expected success, but got an error: <errors.errorString \| 0xc000ecb3d0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.004s. Expected success, but got an error: <errors.errorString \| 0xc000ecb3d0>: missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:16 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f3a168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-tcvs9s" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007c8f20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-tcvs9s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007c8f00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-tcvs9s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007c8ee0>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-tcvs9s\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f3a168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-tcvs9s" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007c8f20>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-tcvs9s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007c8f00>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-tcvs9s\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007c8ee0>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-tcvs9s\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:16 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dee4c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-spbt5k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008fe700>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-spbt5k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008fe6e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-spbt5k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e0ee20>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-spbt5k\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000dee4c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-spbt5k" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0008fe700>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-spbt5k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0008fe6e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-spbt5k\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000e0ee20>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-spbt5k\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:16 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00140e120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-24h8dp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001490da0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-24h8dp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001490d80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-24h8dp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001490d60>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-24h8dp\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00140e120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-24h8dp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001490da0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-24h8dp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001490d80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-24h8dp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001490d60>{ msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-24h8dp\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:16 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0000130c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-mqv2gt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000719300>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-mqv2gt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007192e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-mqv2gt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007192a0>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-mqv2gt\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0000130c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-mqv2gt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000719300>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-mqv2gt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007192e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-mqv2gt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007192a0>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-mqv2gt\" to finish deleting: context canceled", err: <errors.errorString \| 0xae100d0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:37:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 0ffc4b6 · merge 532823b	n/a	e2e	failed	★#4468 (merged)	15	e2e (15) failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:08 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00149c150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-w2cn9m" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000703960>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-w2cn9m\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000703900>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-w2cn9m\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007038e0>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-w2cn9m\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00149c150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-w2cn9m" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000703960>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-w2cn9m\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000703900>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-w2cn9m\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007038e0>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-w2cn9m\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:08 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc000ebece0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-ptdfmd" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb", "name": "cb5598e6-3412-4c1e-88db-7cb59ed1e7eb", "status": "Failed", "startTime": "2026-03-16T12:53:51.06924507Z", "endTime": "2026-03-16T12:54:44.068464049Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-ptdfmd\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\",\n \"name\": \"cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:53:51.06924507Z\",\n \"endTime\": \"2026-03-16T12:54:44.068464049Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000ebecc0>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-ptdfmd\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\",\n \"name\": \"cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:53:51.06924507Z\",\n \"endTime\": \"2026-03-16T12:54:44.068464049Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc000ebece0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-ptdfmd" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb", "name": "cb5598e6-3412-4c1e-88db-7cb59ed1e7eb", "status": "Failed", "startTime": "2026-03-16T12:53:51.06924507Z", "endTime": "2026-03-16T12:54:44.068464049Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-ptdfmd\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\",\n \"name\": \"cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:53:51.06924507Z\",\n \"endTime\": \"2026-03-16T12:54:44.068464049Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000ebecc0>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-ptdfmd\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\",\n \"name\": \"cb5598e6-3412-4c1e-88db-7cb59ed1e7eb\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:53:51.06924507Z\",\n \"endTime\": \"2026-03-16T12:54:44.068464049Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred client rate limiter Wait returned an error: context deadline exceeded 11:37:08 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc000d9b548>: context deadline exceeded failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded { errs: [ <context.deadlineExceededError>{}, <fmt.wrapError \| 0xc000492760>{ msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded", err: <fmt.wrapError \| 0xc000492740>{ msg: "client rate limiter Wait returned an error: context deadline exceeded", err: <context.deadlineExceededError>{}, }, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.joinError \| 0xc000d9b548>: context deadline exceeded failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded { errs: [ <context.deadlineExceededError>{}, <fmt.wrapError \| 0xc000492760>{ msg: "failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded", err: <fmt.wrapError \| 0xc000492740>{ msg: "client rate limiter Wait returned an error: context deadline exceeded", err: <context.deadlineExceededError>{}, }, }, ], } occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:08 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00136e120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-zpjlr5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001011580>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-zpjlr5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001011560>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-zpjlr5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001011540>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-zpjlr5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00136e120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-zpjlr5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001011580>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-zpjlr5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001011560>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-zpjlr5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001011540>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-zpjlr5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:08 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00116c498>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-7pvp62" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00065a660>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-7pvp62\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00065a640>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-7pvp62\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f8c7c0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-7pvp62\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00116c498>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-7pvp62" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00065a660>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-7pvp62\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00065a640>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-7pvp62\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f8c7c0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-7pvp62\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:08 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc0009c0340>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kb7d8n" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d", "name": "185fa830-1f51-4ccd-83ef-db03686add8d", "status": "Failed", "startTime": "2026-03-16T12:49:01.211413542Z", "endTime": "2026-03-16T12:50:00.809428459Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-kb7d8n\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d\",\n \"name\": \"185fa830-1f51-4ccd-83ef-db03686add8d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:01.211413542Z\",\n \"endTime\": \"2026-03-16T12:50:00.809428459Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0009c0320>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-kb7d8n\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d\",\n \"name\": \"185fa830-1f51-4ccd-83ef-db03686add8d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:01.211413542Z\",\n \"endTime\": \"2026-03-16T12:50:00.809428459Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc0009c0340>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kb7d8n" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d", "name": "185fa830-1f51-4ccd-83ef-db03686add8d", "status": "Failed", "startTime": "2026-03-16T12:49:01.211413542Z", "endTime": "2026-03-16T12:50:00.809428459Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-kb7d8n\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d\",\n \"name\": \"185fa830-1f51-4ccd-83ef-db03686add8d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:01.211413542Z\",\n \"endTime\": \"2026-03-16T12:50:00.809428459Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0009c0320>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-kb7d8n\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/185fa830-1f51-4ccd-83ef-db03686add8d\",\n \"name\": \"185fa830-1f51-4ccd-83ef-db03686add8d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:01.211413542Z\",\n \"endTime\": \"2026-03-16T12:50:00.809428459Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-66/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:08 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014281f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listm9jkb8" in resourcegroup="ea-list-255dgx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00140d080>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listm9jkb8\" in resourcegroup=\"ea-list-255dgx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00140d060>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listm9jkb8\" in resourcegroup=\"ea-list-255dgx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00140d040>{ msg: "failed waiting for hcpCluster=\"ea-listm9jkb8\" in resourcegroup=\"ea-list-255dgx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014281f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listm9jkb8" in resourcegroup="ea-list-255dgx" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00140d080>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listm9jkb8\" in resourcegroup=\"ea-list-255dgx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00140d060>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-listm9jkb8\" in resourcegroup=\"ea-list-255dgx\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00140d040>{ msg: "failed waiting for hcpCluster=\"ea-listm9jkb8\" in resourcegroup=\"ea-list-255dgx\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred missing expected log sources: [ocm-arohcp/kube-apiserver] 11:37:08 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc001287180>: missing expected log sources: [ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc001287180>: missing expected log sources: [ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:08 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014641e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-gqtm95" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000800040>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-gqtm95\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000800020>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-gqtm95\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007c1f60>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-gqtm95\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014641e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-gqtm95" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000800040>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-gqtm95\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000800020>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-gqtm95\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007c1f60>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-gqtm95\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:08 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001048300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-pvkkz9" in resourcegroup="admin-credential-lifecycle-test-t5rxqt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00084b400>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-pvkkz9\" in resourcegroup=\"admin-credential-lifecycle-test-t5rxqt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00084b3e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-pvkkz9\" in resourcegroup=\"admin-credential-lifecycle-test-t5rxqt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00084b3c0>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-pvkkz9\" in resourcegroup=\"admin-credential-lifecycle-test-t5rxqt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001048300>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-pvkkz9" in resourcegroup="admin-credential-lifecycle-test-t5rxqt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00084b400>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-pvkkz9\" in resourcegroup=\"admin-credential-lifecycle-test-t5rxqt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00084b3e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-pvkkz9\" in resourcegroup=\"admin-credential-lifecycle-test-t5rxqt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00084b3c0>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-pvkkz9\" in resourcegroup=\"admin-credential-lifecycle-test-t5rxqt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:08 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014481b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-tk2rlb" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001500f60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tk2rlb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001500f40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tk2rlb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001500f20>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tk2rlb\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0014481b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-tk2rlb" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001500f60>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tk2rlb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001500f40>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tk2rlb\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001500f20>{ msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-tk2rlb\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009ed3b0>: failed to cleanup resource group: failed waiting for resourcegroup="clusternp128-4dtxfh" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0000deb20>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"clusternp128-4dtxfh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0000deb00>{ msg: "failed waiting for resourcegroup=\"clusternp128-4dtxfh\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0009ed3b0>: failed to cleanup resource group: failed waiting for resourcegroup="clusternp128-4dtxfh" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0000deb20>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"clusternp128-4dtxfh\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0000deb00>{ msg: "failed waiting for resourcegroup=\"clusternp128-4dtxfh\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:08 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000a4a880>: failed to create HCP cluster cluster-zstream-4-20-cdpvqt: failed waiting for cluster="cluster-zstream-4-20-cdpvqt" in resourcegroup="rg-zstream-upgrade-4-20-bccpt6" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b", "name": "c51f19aa-b670-4113-9baa-6af8f159532b", "status": "Failed", "startTime": "2026-03-16T12:49:02.264347005Z", "endTime": "2026-03-16T12:50:00.815626772Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-cdpvqt: failed waiting for cluster=\"cluster-zstream-4-20-cdpvqt\" in resourcegroup=\"rg-zstream-upgrade-4-20-bccpt6\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b\",\n \"name\": \"c51f19aa-b670-4113-9baa-6af8f159532b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:02.264347005Z\",\n \"endTime\": \"2026-03-16T12:50:00.815626772Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a4a860>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-cdpvqt\" in resourcegroup=\"rg-zstream-upgrade-4-20-bccpt6\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b\",\n \"name\": \"c51f19aa-b670-4113-9baa-6af8f159532b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:02.264347005Z\",\n \"endTime\": \"2026-03-16T12:50:00.815626772Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subsc... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000a4a880>: failed to create HCP cluster cluster-zstream-4-20-cdpvqt: failed waiting for cluster="cluster-zstream-4-20-cdpvqt" in resourcegroup="rg-zstream-upgrade-4-20-bccpt6" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b", "name": "c51f19aa-b670-4113-9baa-6af8f159532b", "status": "Failed", "startTime": "2026-03-16T12:49:02.264347005Z", "endTime": "2026-03-16T12:50:00.815626772Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-cdpvqt: failed waiting for cluster=\"cluster-zstream-4-20-cdpvqt\" in resourcegroup=\"rg-zstream-upgrade-4-20-bccpt6\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b\",\n \"name\": \"c51f19aa-b670-4113-9baa-6af8f159532b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:02.264347005Z\",\n \"endTime\": \"2026-03-16T12:50:00.815626772Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a4a860>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-cdpvqt\" in resourcegroup=\"rg-zstream-upgrade-4-20-bccpt6\" to finish creating: GET https://rp.j2651136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c51f19aa-b670-4113-9baa-6af8f159532b\",\n \"name\": \"c51f19aa-b670-4113-9baa-6af8f159532b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:49:02.264347005Z\",\n \"endTime\": \"2026-03-16T12:50:00.815626772Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-138/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: Subsc... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010b5dd0>: failed to cleanup resource group: failed waiting for resourcegroup="pullsecret-test-bhtgnp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000693200>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"pullsecret-test-bhtgnp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006931c0>{ msg: "failed waiting for resourcegroup=\"pullsecret-test-bhtgnp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0010b5dd0>: failed to cleanup resource group: failed waiting for resourcegroup="pullsecret-test-bhtgnp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000693200>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"pullsecret-test-bhtgnp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0006931c0>{ msg: "failed waiting for resourcegroup=\"pullsecret-test-bhtgnp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f36690>: failed to cleanup resource group: failed waiting for resourcegroup="autoscaling-cluster-2bvkgc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000857c60>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"autoscaling-cluster-2bvkgc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000857c40>{ msg: "failed waiting for resourcegroup=\"autoscaling-cluster-2bvkgc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f36690>: failed to cleanup resource group: failed waiting for resourcegroup="autoscaling-cluster-2bvkgc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000857c60>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"autoscaling-cluster-2bvkgc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000857c40>{ msg: "failed waiting for resourcegroup=\"autoscaling-cluster-2bvkgc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:37:01	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head dfb6412	n/a	e2e	failed	★#4470 (open)	18	e2e (18) ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:01 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapError \| 0xc000e88380>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-mrbvrt" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03", "name": "625b7db7-c87b-4386-b2f9-bc83bcf11a03", "status": "Failed", "startTime": "2026-03-16T12:40:58.976812987Z", "endTime": "2026-03-16T12:41:58.144975207Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-mrbvrt\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03\",\n \"name\": \"625b7db7-c87b-4386-b2f9-bc83bcf11a03\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:58.976812987Z\",\n \"endTime\": \"2026-03-16T12:41:58.144975207Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e88280>{ msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-mrbvrt\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03\",\n \"name\": \"625b7db7-c87b-4386-b2f9-bc83bcf11a03\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:58.976812987Z\",\n \"endTime\": \"2026-03-16T12:41:58.144975207Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"e... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapError \| 0xc000e88380>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-mrbvrt" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03", "name": "625b7db7-c87b-4386-b2f9-bc83bcf11a03", "status": "Failed", "startTime": "2026-03-16T12:40:58.976812987Z", "endTime": "2026-03-16T12:41:58.144975207Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-mrbvrt\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03\",\n \"name\": \"625b7db7-c87b-4386-b2f9-bc83bcf11a03\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:58.976812987Z\",\n \"endTime\": \"2026-03-16T12:41:58.144975207Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e88280>{ msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-mrbvrt\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/625b7db7-c87b-4386-b2f9-bc83bcf11a03\",\n \"name\": \"625b7db7-c87b-4386-b2f9-bc83bcf11a03\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:58.976812987Z\",\n \"endTime\": \"2026-03-16T12:41:58.144975207Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-222/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"e... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:01 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc000d870e0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-tbnv77" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de", "name": "aa903328-1672-4406-a5b2-4a1df3f220de", "status": "Failed", "startTime": "2026-03-16T12:45:59.548150295Z", "endTime": "2026-03-16T12:48:12.416287908Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tbnv77\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de\",\n \"name\": \"aa903328-1672-4406-a5b2-4a1df3f220de\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:45:59.548150295Z\",\n \"endTime\": \"2026-03-16T12:48:12.416287908Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000d870c0>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tbnv77\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de\",\n \"name\": \"aa903328-1672-4406-a5b2-4a1df3f220de\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:45:59.548150295Z\",\n \"endTime\": \"2026-03-16T12:48:12.416287908Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc000d870e0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-tbnv77" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de", "name": "aa903328-1672-4406-a5b2-4a1df3f220de", "status": "Failed", "startTime": "2026-03-16T12:45:59.548150295Z", "endTime": "2026-03-16T12:48:12.416287908Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tbnv77\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de\",\n \"name\": \"aa903328-1672-4406-a5b2-4a1df3f220de\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:45:59.548150295Z\",\n \"endTime\": \"2026-03-16T12:48:12.416287908Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '2' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000d870c0>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-tbnv77\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa903328-1672-4406-a5b2-4a1df3f220de\",\n \"name\": \"aa903328-1672-4406-a5b2-4a1df3f220de\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:45:59.548150295Z\",\n \"endTime\": \"2026-03-16T12:48:12.416287908Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift 11:37:01 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapError \| 0xc00136c380>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9jsvk2" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/120e5a0f-75bd-4759-8ece-4c9c94be6614 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/120e5a0f-75bd-4759-8ece-4c9c94be6614", "name": "120e5a0f-75bd-4759-8ece-4c9c94be6614", "status": "Failed", "startTime": "2026-03-16T12:40:57.249642697Z", "endTime": "2026-03-16T12:41:58.288273824Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get network security group 'arm64-vm-customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9jsvk2/providers/Microsoft.Network/networkSecurityGroups/arm64-vm-customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-9jsvk2\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/120e5a0f-75bd-4759-8ece-4c9c94be6614\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/120e5a0f-75bd-4759-8ece-4c9c94be6614\",\n \"name\": \"120e5a0f-75bd-4759-8ece-4c9c94be6614\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:57.249642697Z\",\n \"endTime\": \"2026-03-16T12:41:58.288273824Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'arm64-vm-customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9jsvk2/providers/Microsoft.Network/networkSecurityGroups/arm64-vm-customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00136c360>{ msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-9jsvk2\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/120e5a0f-75bd-4759-8ece-4c9c94be6614\n---------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapError \| 0xc00136c380>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9jsvk2" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/120e5a0f-75bd-4759-8ece-4c9c94be6614 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/120e5a0f-75bd-4759-8ece-4c9c94be6614", "name": "120e5a0f-75bd-4759-8ece-4c9c94be6614", "status": "Failed", "startTime": "2026-03-16T12:40:57.249642697Z", "endTime": "2026-03-16T12:41:58.288273824Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" }, { "code": "InternalServerError", "message": "failed to get network security group 'arm64-vm-customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9jsvk2/providers/Microsoft.Network/networkSecurityGroups/arm64-vm-customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-9jsvk2\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/120e5a0f-75bd-4759-8ece-4c9c94be6614\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/120e5a0f-75bd-4759-8ece-4c9c94be6614\",\n \"name\": \"120e5a0f-75bd-4759-8ece-4c9c94be6614\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:57.249642697Z\",\n \"endTime\": \"2026-03-16T12:41:58.288273824Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'arm64-vm-customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9jsvk2/providers/Microsoft.Network/networkSecurityGroups/arm64-vm-customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00136c360>{ msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-9jsvk2\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/120e5a0f-75bd-4759-8ece-4c9c94be6614\n---------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ede8d0>: failed to cleanup resource group: failed waiting for resourcegroup="autoscaling-cluster-9x6bsc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001399540>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"autoscaling-cluster-9x6bsc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001399520>{ msg: "failed waiting for resourcegroup=\"autoscaling-cluster-9x6bsc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ede8d0>: failed to cleanup resource group: failed waiting for resourcegroup="autoscaling-cluster-9x6bsc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001399540>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"autoscaling-cluster-9x6bsc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001399520>{ msg: "failed waiting for resourcegroup=\"autoscaling-cluster-9x6bsc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:01 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00111a150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-xw5rxt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000176640>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-xw5rxt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000176600>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-xw5rxt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0001765c0>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-xw5rxt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00111a150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-xw5rxt" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000176640>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-xw5rxt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000176600>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-xw5rxt\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0001765c0>{ msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-xw5rxt\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed 11:37:01 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:152]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:152]: Cluster provisioning failed failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0012de1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-j4dj27" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ddd9c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-j4dj27\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ddd980>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-j4dj27\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ddd940>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-j4dj27\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0012de1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-j4dj27" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000ddd9c0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-j4dj27\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ddd980>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-j4dj27\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000ddd940>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-j4dj27\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:01 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000c4c360>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-r2jsvz" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d", "name": "577c0a8d-dcc6-4d43-97f0-70c20b633e7d", "status": "Failed", "startTime": "2026-03-16T12:41:08.167401696Z", "endTime": "2026-03-16T12:41:58.502094725Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-r2jsvz\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d\",\n \"name\": \"577c0a8d-dcc6-4d43-97f0-70c20b633e7d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:41:08.167401696Z\",\n \"endTime\": \"2026-03-16T12:41:58.502094725Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c4c340>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-r2jsvz\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d\",\n \"name\": \"577c0a8d-dcc6-4d43-97f0-70c20b633e7d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:41:08.167401696Z\",\n \"endTime\": \"2026-03-16T12:41:58.502094725Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000c4c360>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-r2jsvz" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d", "name": "577c0a8d-dcc6-4d43-97f0-70c20b633e7d", "status": "Failed", "startTime": "2026-03-16T12:41:08.167401696Z", "endTime": "2026-03-16T12:41:58.502094725Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-r2jsvz\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d\",\n \"name\": \"577c0a8d-dcc6-4d43-97f0-70c20b633e7d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:41:08.167401696Z\",\n \"endTime\": \"2026-03-16T12:41:58.502094725Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c4c340>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-r2jsvz\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/577c0a8d-dcc6-4d43-97f0-70c20b633e7d\",\n \"name\": \"577c0a8d-dcc6-4d43-97f0-70c20b633e7d\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:41:08.167401696Z\",\n \"endTime\": \"2026-03-16T12:41:58.502094725Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-263/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:01 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error: <exported.ResponseError \| 0xc0004cfe00>: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2bf8b42e-2ebc-476b-b395-b387e0daa355 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2bf8b42e-2ebc-476b-b395-b387e0daa355", "name": "2bf8b42e-2ebc-476b-b395-b387e0daa355", "status": "Failed", "startTime": "2026-03-16T12:41:56.025534294Z", "endTime": "2026-03-16T12:42:51.560044029Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-251/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-251/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "997db521-3d5d-46d8-b08f-8b1664aa2128", ], "Date": [ "Mon, 16 Mar 2026 12:42:56 GMT", ], "Content-Length": ["1565"], "X-Envoy-Upstream-Service-Time": ["8"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000a91140>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 50, 98, 102, 56, 98, 52, 50, 101, 45, 50, 101, 98, 99, 45, 52, 55, 54, 98, 45, 98, 51, 57, 53, 45, 98, 51, 56, 55, 101, 48, 100, 97, 97, 51, 53, 53, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 50, 98, 102, 56, 98, 52, 50, 101, 45, 50, 101, 98, 99, 45, 52, 55, 54, 98, 45, 98, 51, 57, 53, 45, 98, 51, 56, 55, 101, 48, 100, 97, 97, 51, 53, 53, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 50, 58, 52, 49, 58, 53, 54, 46, 48, 50, 53, 53, 51, 52, 50, 57, 52, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 50, 58, 52, 50, 58, 53, 49, 46, 53, 54, 48, 48, 52, 52, 48, 50, 57, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 50, 53, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 100, 105, 115, 107, 45, 99, 115, 105, 45, 100, 114, 105, 118, 101, 114, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error: <exported.ResponseError \| 0xc0004cfe00>: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2bf8b42e-2ebc-476b-b395-b387e0daa355 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2bf8b42e-2ebc-476b-b395-b387e0daa355", "name": "2bf8b42e-2ebc-476b-b395-b387e0daa355", "status": "Failed", "startTime": "2026-03-16T12:41:56.025534294Z", "endTime": "2026-03-16T12:42:51.560044029Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-251/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-251/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "997db521-3d5d-46d8-b08f-8b1664aa2128", ], "Date": [ "Mon, 16 Mar 2026 12:42:56 GMT", ], "Content-Length": ["1565"], "X-Envoy-Upstream-Service-Time": ["8"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000a91140>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 50, 98, 102, 56, 98, 52, 50, 101, 45, 50, 101, 98, 99, 45, 52, 55, 54, 98, 45, 98, 51, 57, 53, 45, 98, 51, 56, 55, 101, 48, 100, 97, 97, 51, 53, 53, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 50, 98, 102, 56, 98, 52, 50, 101, 45, 50, 101, 98, 99, 45, 52, 55, 54, 98, 45, 98, 51, 57, 53, 45, 98, 51, 56, 55, 101, 48, 100, 97, 97, 51, 53, 53, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 50, 58, 52, 49, 58, 53, 54, 46, 48, 50, 53, 53, 51, 52, 50, 57, 52, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 54, 84, 49, 50, 58, 52, 50, 58, 53, 49, 46, 53, 54, 48, 48, 52, 52, 48, 50, 57, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 50, 53, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 100, 105, 115, 107, 45, 99, 115, 105, 45, 100, 114, 105, 118, 101, 114, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred missing expected log sources: [ocm-arohcp/kube-apiserver] 11:37:01 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc001751a60>: missing expected log sources: [ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.009s. Expected success, but got an error: <errors.errorString \| 0xc001751a60>: missing expected log sources: [ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:01 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d89cb0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-19-7nzxrw" in resourcegroup="rg-zstream-upgrade-4-19-pfzszd" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a580a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-7nzxrw\" in resourcegroup=\"rg-zstream-upgrade-4-19-pfzszd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a58080>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-7nzxrw\" in resourcegroup=\"rg-zstream-upgrade-4-19-pfzszd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a58060>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-7nzxrw\" in resourcegroup=\"rg-zstream-upgrade-4-19-pfzszd\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000d89cb0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-19-7nzxrw" in resourcegroup="rg-zstream-upgrade-4-19-pfzszd" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a580a0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-7nzxrw\" in resourcegroup=\"rg-zstream-upgrade-4-19-pfzszd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a58080>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-zstream-4-19-7nzxrw\" in resourcegroup=\"rg-zstream-upgrade-4-19-pfzszd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a58060>{ msg: "failed waiting for hcpCluster=\"cluster-zstream-4-19-7nzxrw\" in resourcegroup=\"rg-zstream-upgrade-4-19-pfzszd\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:01 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f50e70>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-hjsdwz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007e7ca0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-hjsdwz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007e7c80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-hjsdwz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007e7c60>{ msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-hjsdwz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f50e70>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-hjsdwz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0007e7ca0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-hjsdwz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007e7c80>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-hjsdwz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0007e7c60>{ msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-hjsdwz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:37:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ee81c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-t49wwc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0004a6620>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-t49wwc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0004a6600>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-t49wwc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0004a6520>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-t49wwc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ee81c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-t49wwc" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0004a6620>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-t49wwc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0004a6600>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-t49wwc\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0004a6520>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-t49wwc\" to finish deleting: context canceled", err: <errors.errorString \| 0xb09b0c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:01 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc000e84d60>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-zdjk7d" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16", "name": "74f07a94-3ad2-43f0-b896-ab9ac9199f16", "status": "Failed", "startTime": "2026-03-16T12:43:33.474385131Z", "endTime": "2026-03-16T12:44:39.436485773Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-zdjk7d\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16\",\n \"name\": \"74f07a94-3ad2-43f0-b896-ab9ac9199f16\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:43:33.474385131Z\",\n \"endTime\": \"2026-03-16T12:44:39.436485773Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e84d40>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-zdjk7d\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16\",\n \"name\": \"74f07a94-3ad2-43f0-b896-ab9ac9199f16\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:43:33.474385131Z\",\n \"endTime\": \"2026-03-16T12:44:39.436485773Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc000e84d60>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-zdjk7d" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16", "name": "74f07a94-3ad2-43f0-b896-ab9ac9199f16", "status": "Failed", "startTime": "2026-03-16T12:43:33.474385131Z", "endTime": "2026-03-16T12:44:39.436485773Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-zdjk7d\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16\",\n \"name\": \"74f07a94-3ad2-43f0-b896-ab9ac9199f16\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:43:33.474385131Z\",\n \"endTime\": \"2026-03-16T12:44:39.436485773Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e84d40>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-zdjk7d\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74f07a94-3ad2-43f0-b896-ab9ac9199f16\",\n \"name\": \"74f07a94-3ad2-43f0-b896-ab9ac9199f16\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:43:33.474385131Z\",\n \"endTime\": \"2026-03-16T12:44:39.436485773Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-159/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapError \| 0xc000f85740>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-pxgn8m" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a", "name": "67188cb4-37b8-4ab8-b4f7-1402512d327a", "status": "Failed", "startTime": "2026-03-16T12:40:59.142803014Z", "endTime": "2026-03-16T12:41:58.467762398Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pxgn8m\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a\",\n \"name\": \"67188cb4-37b8-4ab8-b4f7-1402512d327a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:59.142803014Z\",\n \"endTime\": \"2026-03-16T12:41:58.467762398Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f85720>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pxgn8m\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a\",\n \"name\": \"67188cb4-37b8-4ab8-b4f7-1402512d327a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:59.142803014Z\",\n \"endTime\": \"2026-03-16T12:41:58.467762398Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapError \| 0xc000f85740>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-pxgn8m" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a", "name": "67188cb4-37b8-4ab8-b4f7-1402512d327a", "status": "Failed", "startTime": "2026-03-16T12:40:59.142803014Z", "endTime": "2026-03-16T12:41:58.467762398Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pxgn8m\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a\",\n \"name\": \"67188cb4-37b8-4ab8-b4f7-1402512d327a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:59.142803014Z\",\n \"endTime\": \"2026-03-16T12:41:58.467762398Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f85720>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pxgn8m\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67188cb4-37b8-4ab8-b4f7-1402512d327a\",\n \"name\": \"67188cb4-37b8-4ab8-b4f7-1402512d327a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:59.142803014Z\",\n \"endTime\": \"2026-03-16T12:41:58.467762398Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-273/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:01 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000db0440>: failed to create HCP cluster cluster-zstream-4-21-tjnwkc: failed waiting for cluster="cluster-zstream-4-21-tjnwkc" in resourcegroup="rg-zstream-upgrade-4-21-xgkq8d" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9", "name": "94d48c6b-47fa-4a94-9a31-f62dfadfe9a9", "status": "Failed", "startTime": "2026-03-16T12:46:09.542833596Z", "endTime": "2026-03-16T12:47:06.512450556Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-tjnwkc: failed waiting for cluster=\"cluster-zstream-4-21-tjnwkc\" in resourcegroup=\"rg-zstream-upgrade-4-21-xgkq8d\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\",\n \"name\": \"94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:46:09.542833596Z\",\n \"endTime\": \"2026-03-16T12:47:06.512450556Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000db0420>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-tjnwkc\" in resourcegroup=\"rg-zstream-upgrade-4-21-xgkq8d\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\",\n \"name\": \"94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:46:09.542833596Z\",\n \"endTime\": \"2026-03-16T12:47:06.512450556Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000db0440>: failed to create HCP cluster cluster-zstream-4-21-tjnwkc: failed waiting for cluster="cluster-zstream-4-21-tjnwkc" in resourcegroup="rg-zstream-upgrade-4-21-xgkq8d" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9", "name": "94d48c6b-47fa-4a94-9a31-f62dfadfe9a9", "status": "Failed", "startTime": "2026-03-16T12:46:09.542833596Z", "endTime": "2026-03-16T12:47:06.512450556Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-tjnwkc: failed waiting for cluster=\"cluster-zstream-4-21-tjnwkc\" in resourcegroup=\"rg-zstream-upgrade-4-21-xgkq8d\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\",\n \"name\": \"94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:46:09.542833596Z\",\n \"endTime\": \"2026-03-16T12:47:06.512450556Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000db0420>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-tjnwkc\" in resourcegroup=\"rg-zstream-upgrade-4-21-xgkq8d\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\",\n \"name\": \"94d48c6b-47fa-4a94-9a31-f62dfadfe9a9\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:46:09.542833596Z\",\n \"endTime\": \"2026-03-16T12:47:06.512450556Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-197/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:37:01 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc0008c4680>: failed to create HCP cluster ea-listm2rzdz: failed waiting for cluster="ea-listm2rzdz" in resourcegroup="ea-list-2tsj85" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf", "name": "da6244ff-1caa-4262-a5b2-f747dd758fcf", "status": "Failed", "startTime": "2026-03-16T12:40:57.66091959Z", "endTime": "2026-03-16T12:41:58.815134562Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listm2rzdz: failed waiting for cluster=\"ea-listm2rzdz\" in resourcegroup=\"ea-list-2tsj85\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf\",\n \"name\": \"da6244ff-1caa-4262-a5b2-f747dd758fcf\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:57.66091959Z\",\n \"endTime\": \"2026-03-16T12:41:58.815134562Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008c4660>{ msg: "failed waiting for cluster=\"ea-listm2rzdz\" in resourcegroup=\"ea-list-2tsj85\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf\",\n \"name\": \"da6244ff-1caa-4262-a5b2-f747dd758fcf\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:57.66091959Z\",\n \"endTime\": \"2026-03-16T12:41:58.815134562Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc0008c4680>: failed to create HCP cluster ea-listm2rzdz: failed waiting for cluster="ea-listm2rzdz" in resourcegroup="ea-list-2tsj85" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf", "name": "da6244ff-1caa-4262-a5b2-f747dd758fcf", "status": "Failed", "startTime": "2026-03-16T12:40:57.66091959Z", "endTime": "2026-03-16T12:41:58.815134562Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listm2rzdz: failed waiting for cluster=\"ea-listm2rzdz\" in resourcegroup=\"ea-list-2tsj85\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf\",\n \"name\": \"da6244ff-1caa-4262-a5b2-f747dd758fcf\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:57.66091959Z\",\n \"endTime\": \"2026-03-16T12:41:58.815134562Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008c4660>{ msg: "failed waiting for cluster=\"ea-listm2rzdz\" in resourcegroup=\"ea-list-2tsj85\" to finish creating: GET https://rp.j7220480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da6244ff-1caa-4262-a5b2-f747dd758fcf\",\n \"name\": \"da6244ff-1caa-4262-a5b2-f747dd758fcf\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:40:57.66091959Z\",\n \"endTime\": \"2026-03-16T12:41:58.815134562Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-283/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] 11:37:01 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.000s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Expected <[]string \| len:1, cap:1>: [ "image-registry (Available=False)", ] to be empty fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.000s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [image-registry (Available=False)] Expected <[]string \| len:1, cap:1>: [ "image-registry (Available=False)", ] to be empty
11:36:53	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head bec5815 · merge 8b7576c	n/a	unknown	failed	#4473 (merged)	1	non-artifact-backed failure (no junit artifacts)
11:36:52	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7996885	n/a	unknown	failed	#4477 (open)	1	non-artifact-backed failure (no junit artifacts)
11:36:44	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e0b9c94 · merge e0b9c94	n/a	e2e	failed	★#4478 (merged)	15	e2e (15) ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity 11:36:44 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapError \| 0xc0005a0660>: failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-pslrw6" to finish creating: GET https://rp.j7248128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af", "name": "2cb08497-c1a3-40ae-8441-14884f9719af", "status": "Failed", "startTime": "2026-03-16T12:39:38.97159587Z", "endTime": "2026-03-16T12:41:03.165372775Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-pslrw6\" to finish creating: GET https://rp.j7248128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af\",\n \"name\": \"2cb08497-c1a3-40ae-8441-14884f9719af\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:38.97159587Z\",\n \"endTime\": \"2026-03-16T12:41:03.165372775Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005a0640>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-pslrw6\" to finish creating: GET https://rp.j7248128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af\",\n \"name\": \"2cb08497-c1a3-40ae-8441-14884f9719af\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:38.97159587Z\",\n \"endTime\": \"2026-03-16T12:41:03.165372775Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapError \| 0xc0005a0660>: failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-pslrw6" to finish creating: GET https://rp.j7248128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af", "name": "2cb08497-c1a3-40ae-8441-14884f9719af", "status": "Failed", "startTime": "2026-03-16T12:39:38.97159587Z", "endTime": "2026-03-16T12:41:03.165372775Z", "error": { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"SubscriptionRequestsThrottled\",\n \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-pslrw6\" to finish creating: GET https://rp.j7248128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af\",\n \"name\": \"2cb08497-c1a3-40ae-8441-14884f9719af\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:38.97159587Z\",\n \"endTime\": \"2026-03-16T12:41:03.165372775Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005a0640>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-pslrw6\" to finish creating: GET https://rp.j7248128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2cb08497-c1a3-40ae-8441-14884f9719af\",\n \"name\": \"2cb08497-c1a3-40ae-8441-14884f9719af\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T12:39:38.97159587Z\",\n \"endTime\": \"2026-03-16T12:41:03.165372775Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-215/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-----------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f2c4f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-55lxx7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0011f5400>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-55lxx7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011f53e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-55lxx7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011f53c0>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-55lxx7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f2c4f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-55lxx7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0011f5400>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-55lxx7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011f53e0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-55lxx7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011f53c0>{ msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-55lxx7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: failed waiting for resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001198570>: failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-7ftvnr" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000801440>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-7ftvnr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000801420>{ msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-7ftvnr\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001198570>: failed to cleanup resource group: failed waiting for resourcegroup="e2e-cidr-connectivity-7ftvnr" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000801440>{ msg: "failed to cleanup resource group: failed waiting for resourcegroup=\"e2e-cidr-connectivity-7ftvnr\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000801420>{ msg: "failed waiting for resourcegroup=\"e2e-cidr-connectivity-7ftvnr\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ec2858>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-hr9wkd" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000721980>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-hr9wkd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000721960>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-hr9wkd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00154e9e0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-hr9wkd\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000ec2858>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-hr9wkd" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000721980>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-hr9wkd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000721960>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-hr9wkd\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00154e9e0>{ msg: "failed waiting for hcpCluster=\"basic-hcp-cluster2\" in resourcegroup=\"customer-rg-hr9wkd\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001500198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-9dw6pw" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001802e40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9dw6pw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001802e20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9dw6pw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001802e00>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9dw6pw\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc001500198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-9dw6pw" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001802e40>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9dw6pw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001802e20>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9dw6pw\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001802e00>{ msg: "failed waiting for hcpCluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9dw6pw\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00124c1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-x9wqmp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0005a1ee0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-x9wqmp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0005a1ec0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-x9wqmp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0005a1ea0>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-x9wqmp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00124c1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-x9wqmp" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0005a1ee0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-x9wqmp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0005a1ec0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-x9wqmp\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0005a1ea0>{ msg: "failed waiting for hcpCluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-x9wqmp\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000eee450>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-c45gh9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001193220>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-c45gh9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001193200>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-c45gh9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011931e0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-c45gh9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000eee450>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-c45gh9" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc001193220>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-c45gh9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc001193200>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-c45gh9\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0011931e0>{ msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-c45gh9\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00079b260>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-l2plxm" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000709160>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-l2plxm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000709140>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-l2plxm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000709120>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-l2plxm\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00079b260>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-l2plxm" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000709160>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-l2plxm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000709140>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-l2plxm\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000709120>{ msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-l2plxm\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fc0330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-psb9ww" in resourcegroup="admin-credential-lifecycle-test-9jj5q7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b8b2e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-psb9ww\" in resourcegroup=\"admin-credential-lifecycle-test-9jj5q7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b8b2c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-psb9ww\" in resourcegroup=\"admin-credential-lifecycle-test-9jj5q7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b8b2a0>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-psb9ww\" in resourcegroup=\"admin-credential-lifecycle-test-9jj5q7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000fc0330>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-psb9ww" in resourcegroup="admin-credential-lifecycle-test-9jj5q7" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b8b2e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-psb9ww\" in resourcegroup=\"admin-credential-lifecycle-test-9jj5q7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b8b2c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-psb9ww\" in resourcegroup=\"admin-credential-lifecycle-test-9jj5q7\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b8b2a0>{ msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-psb9ww\" in resourcegroup=\"admin-credential-lifecycle-test-9jj5q7\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00139c2d0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-q94kvk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0004f68e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-q94kvk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0004f68c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-q94kvk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0004f68a0>{ msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-q94kvk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00139c2d0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-q94kvk" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc0004f68e0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-q94kvk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0004f68c0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-q94kvk\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc0004f68a0>{ msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-q94kvk\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f38198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-85cpq6" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b7eec0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-85cpq6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b7eea0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-85cpq6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b7ee80>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-85cpq6\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000f38198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-85cpq6" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000b7eec0>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-85cpq6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b7eea0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-85cpq6\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000b7ee80>{ msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-85cpq6\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred missing expected log sources: [ocm-arohcp/kube-apiserver] 11:36:44 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s. Expected success, but got an error: <errors.errorString \| 0xc0011e4a70>: missing expected log sources: [ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [ocm-arohcp/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s. Expected success, but got an error: <errors.errorString \| 0xc0011e4a70>: missing expected log sources: [ocm-arohcp/kube-apiserver] { s: "missing expected log sources: [ocm-arohcp/kube-apiserver]", } failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000bf4c30>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-4bgn2f" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f2a180>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4bgn2f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f2a160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4bgn2f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f2a140>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4bgn2f\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc000bf4c30>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-4bgn2f" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000f2a180>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4bgn2f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f2a160>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4bgn2f\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000f2a140>{ msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4bgn2f\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00176a198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-qm2lgz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00090dc00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-qm2lgz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00090dbe0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-qm2lgz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00090dbc0>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-qm2lgz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc00176a198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-qm2lgz" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc00090dc00>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-qm2lgz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00090dbe0>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-qm2lgz\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc00090dbc0>{ msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-qm2lgz\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled 11:36:44 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0013bc450>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-k2vtm5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a1a440>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2vtm5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1a420>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2vtm5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1a400>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2vtm5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited: Unexpected error: <errors.joinError \| 0xc0013bc450>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-k2vtm5" to finish deleting: context canceled { errs: [ <fmt.wrapError \| 0xc000a1a440>{ msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2vtm5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1a420>{ msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2vtm5\" to finish deleting: context canceled", err: <fmt.wrapError \| 0xc000a1a400>{ msg: "failed waiting for hcpCluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-k2vtm5\" to finish deleting: context canceled", err: <errors.errorString \| 0xb0980c0>{ s: "context canceled", }, }, }, }, ], } occurred fail [:0]: A node timeout occurred
11:36:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6b0d466	n/a	unknown	failed	#4479 (open)	1	non-artifact-backed failure (no junit artifacts)
11:00:39	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 2eda048	n/a	unknown	failed	#4117 (closed)	1	non-artifact-backed failure (no junit artifacts)
10:49:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6b0d466	n/a	unknown	failed	#4479 (open)	1	non-artifact-backed failure (no junit artifacts)
10:03:33	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ec4e739	n/a	unknown	failed	#4479 (open)	1	non-artifact-backed failure (no junit artifacts)
08:25:48	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a82c625 · merge a82c625	n/a	n/a	passed	#4347 (merged)	0	n/a
06:33:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head b7f1a9c · merge 205984e	n/a	n/a	passed	#4449 (merged)	0	n/a
06:26:41	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head c45e5a9 · merge ad7003e	n/a	n/a	passed	#4188 (merged)	0	n/a
05:33:57	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7996885	n/a	unknown	failed	#4477 (open)	1	non-artifact-backed failure (no junit artifacts)
04:58:45	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head c98393e · merge c98393e	n/a	n/a	passed	#4476 (merged)	0	n/a
03:27:37	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head bec5815 · merge 8b7576c	n/a	e2e	failed	⚠#4473 (merged)	25	e2e (25) ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc00144fea0>: failed to create HCP cluster ea-listxtfzbl: failed starting cluster creation "ea-listxtfzbl" in resourcegroup="ea-list-jfmwrk": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-jfmwrk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listxtfzbl -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listxtfzbl: failed starting cluster creation \"ea-listxtfzbl\" in resourcegroup=\"ea-list-jfmwrk\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-jfmwrk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listxtfzbl\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00144fe80>{ msg: "failed starting cluster creation \"ea-listxtfzbl\" in resourcegroup=\"ea-list-jfmwrk\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-jfmwrk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listxtfzbl\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001700390>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "8cbaaac2-5d0d-4177-9627-dcb9c5fe7dcf", ], "Date": [ "Mon, 16 Mar 2026 04:15:09 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["31"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc00144fdc0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 50, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc00144fea0>: failed to create HCP cluster ea-listxtfzbl: failed starting cluster creation "ea-listxtfzbl" in resourcegroup="ea-list-jfmwrk": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-jfmwrk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listxtfzbl -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listxtfzbl: failed starting cluster creation \"ea-listxtfzbl\" in resourcegroup=\"ea-list-jfmwrk\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-jfmwrk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listxtfzbl\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00144fe80>{ msg: "failed starting cluster creation \"ea-listxtfzbl\" in resourcegroup=\"ea-list-jfmwrk\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-jfmwrk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listxtfzbl\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001700390>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "8cbaaac2-5d0d-4177-9627-dcb9c5fe7dcf", ], "Date": [ "Mon, 16 Mar 2026 04:15:09 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["31"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc00144fdc0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 50, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapError \| 0xc0005b1ac0>: failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-6xn8gb": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-6xn8gb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6xn8gb\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-6xn8gb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005b1aa0>{ msg: "failed starting cluster creation \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6xn8gb\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-6xn8gb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011bf650>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "3e212603-e8a5-4c0a-abcc-2ae40bf4b97c", ], "Date": [ "Mon, 16 Mar 2026 04:15:52 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["34"], }, Body: <exported.nopClosingBytesReader \| 0xc0005b19e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 3... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapError \| 0xc0005b1ac0>: failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-6xn8gb": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-6xn8gb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6xn8gb\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-6xn8gb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005b1aa0>{ msg: "failed starting cluster creation \"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-6xn8gb\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-6xn8gb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011bf650>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "3e212603-e8a5-4c0a-abcc-2ae40bf4b97c", ], "Date": [ "Mon, 16 Mar 2026 04:15:52 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["34"], }, Body: <exported.nopClosingBytesReader \| 0xc0005b19e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 3... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapError \| 0xc0013d47c0>: failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-j7v8j4": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-j7v8j4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-j7v8j4\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-j7v8j4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0013d47a0>{ msg: "failed starting cluster creation \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-j7v8j4\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-j7v8j4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000e72510>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "3903f69a-5bf4-4b44-afcd-b5a699f79ba1", ], "Date": [ "Mon, 16 Mar 2026 04:14:59 GMT", ], "Content-Length": ["614"], }, Body: <exported.nopClosingBytesReader \| 0xc0013d46e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapError \| 0xc0013d47c0>: failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-j7v8j4": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-j7v8j4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-j7v8j4\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-j7v8j4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0013d47a0>{ msg: "failed starting cluster creation \"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-j7v8j4\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-j7v8j4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000e72510>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "3903f69a-5bf4-4b44-afcd-b5a699f79ba1", ], "Date": [ "Mon, 16 Mar 2026 04:14:59 GMT", ], "Content-Length": ["614"], }, Body: <exported.nopClosingBytesReader \| 0xc0013d46e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapError \| 0xc001026700>: failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-bvplml": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-bvplml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-bvplml\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-bvplml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010266e0>{ msg: "failed starting cluster creation \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-bvplml\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-bvplml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001501e90>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "2dd0563d-27e8-4b59-a5d6-c0ae1ec5ecd7", ], "Date": [ "Mon, 16 Mar 2026 04:15:10 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["30"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc001026620>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapError \| 0xc001026700>: failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-bvplml": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-bvplml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-bvplml\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-bvplml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010266e0>{ msg: "failed starting cluster creation \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-bvplml\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-bvplml/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001501e90>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "2dd0563d-27e8-4b59-a5d6-c0ae1ec5ecd7", ], "Date": [ "Mon, 16 Mar 2026 04:15:10 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["30"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc001026620>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc00082bb20>: failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-vcpd4k": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vcpd4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-vcpd4k\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vcpd4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00082bb00>{ msg: "failed starting cluster creation \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-vcpd4k\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vcpd4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000bc4d80>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "a83e4eff-6581-4d57-8525-7b4553b46f8a", ], "Date": [ "Mon, 16 Mar 2026 04:18:02 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc00082ba40>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc00082bb20>: failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-vcpd4k": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vcpd4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-vcpd4k\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vcpd4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00082bb00>{ msg: "failed starting cluster creation \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-vcpd4k\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-vcpd4k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000bc4d80>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "a83e4eff-6581-4d57-8525-7b4553b46f8a", ], "Date": [ "Mon, 16 Mar 2026 04:18:02 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc00082ba40>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapError \| 0xc0010a5380>: failed to create HCP cluster basic-hcp-cluster: failed starting cluster creation "basic-hcp-cluster" in resourcegroup="basic-cluster-kr6qd2": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-kr6qd2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-hcp-cluster: failed starting cluster creation \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-kr6qd2\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-kr6qd2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010a5360>{ msg: "failed starting cluster creation \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-kr6qd2\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-kr6qd2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00109f920>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "ec42b69e-3ae1-4087-9fc5-1ae1e63e8c3b", ], "Date": [ "Mon, 16 Mar 2026 04:14:38 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["42"], }, Body: <exported.nopClosingBytesReader \| 0xc0010a52a0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapError \| 0xc0010a5380>: failed to create HCP cluster basic-hcp-cluster: failed starting cluster creation "basic-hcp-cluster" in resourcegroup="basic-cluster-kr6qd2": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-kr6qd2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-hcp-cluster: failed starting cluster creation \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-kr6qd2\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-kr6qd2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010a5360>{ msg: "failed starting cluster creation \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-kr6qd2\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-kr6qd2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00109f920>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "ec42b69e-3ae1-4087-9fc5-1ae1e63e8c3b", ], "Date": [ "Mon, 16 Mar 2026 04:14:38 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["42"], }, Body: <exported.nopClosingBytesReader \| 0xc0010a52a0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error: <fmt.wrapError \| 0xc00100a6c0>: failed to create HCP cluster sre-hcp-cluster: failed starting cluster creation "sre-hcp-cluster" in resourcegroup="admin-api-breakglass-zrxld6": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-zrxld6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/sre-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster sre-hcp-cluster: failed starting cluster creation \"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-zrxld6\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-zrxld6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/sre-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00100a6a0>{ msg: "failed starting cluster creation \"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-zrxld6\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-zrxld6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/sre-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000eade90>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["1036"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "f2fd150a-d81c-4682-abff-782da480d223", ], "Date": [ "Mon, 16 Mar 2026 04:15:00 GMT", ], "Content-Length": ["614"], }, Body: <exported.nopClosingBytesReader \| 0xc00100a5e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error: <fmt.wrapError \| 0xc00100a6c0>: failed to create HCP cluster sre-hcp-cluster: failed starting cluster creation "sre-hcp-cluster" in resourcegroup="admin-api-breakglass-zrxld6": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-zrxld6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/sre-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster sre-hcp-cluster: failed starting cluster creation \"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-zrxld6\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-zrxld6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/sre-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00100a6a0>{ msg: "failed starting cluster creation \"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-zrxld6\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-breakglass-zrxld6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/sre-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000eade90>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["1036"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "f2fd150a-d81c-4682-abff-782da480d223", ], "Date": [ "Mon, 16 Mar 2026 04:15:00 GMT", ], "Content-Length": ["614"], }, Body: <exported.nopClosingBytesReader \| 0xc00100a5e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapError \| 0xc0006f8c00>: failed starting cluster creation "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-jhvh92": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-jhvh92/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed starting cluster creation \"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jhvh92\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-jhvh92/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0015190b0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:14:51 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["1037"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "da5ab60c-4711-4b7f-905e-d8a02d0113ae", ], }, Body: <exported.nopClosingBytesReader \| 0xc0006f8aa0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 50, 32, 101, 114, 114, 111, 114, 115, 32, 111, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 100, 101, 116, 97, 105, 108, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 101, 116, 99, 100, 46, 100, 97, 116, 97, 69, 110, 99, 114, 121, 112, 116, 105, 111, 110, 46, 99, 117, 115, 116, 111, 109, 101, 114, 77, 97, 110, 97, 103, 101, 100, 46, 107, 109, 115, 46, 118, 105, 115, 105, 98, 105, 108, 105, 116, 121, 34, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 125, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 1... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapError \| 0xc0006f8c00>: failed starting cluster creation "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-jhvh92": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-jhvh92/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed starting cluster creation \"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-jhvh92\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-jhvh92/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0015190b0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:14:51 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["1037"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "da5ab60c-4711-4b7f-905e-d8a02d0113ae", ], }, Body: <exported.nopClosingBytesReader \| 0xc0006f8aa0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 50, 32, 101, 114, 114, 111, 114, 115, 32, 111, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 100, 101, 116, 97, 105, 108, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 101, 116, 99, 100, 46, 100, 97, 116, 97, 69, 110, 99, 114, 121, 112, 116, 105, 111, 110, 46, 99, 117, 115, 116, 111, 109, 101, 114, 77, 97, 110, 97, 103, 101, 100, 46, 107, 109, 115, 46, 118, 105, 115, 105, 98, 105, 108, 105, 116, 121, 34, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 125, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 1... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error: <fmt.wrapError \| 0xc00077fd80>: failed to create HCP cluster list-test-cluster-l7lj2f: failed starting cluster creation "list-test-cluster-l7lj2f" in resourcegroup="cluster-listing-zpd4tf": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-zpd4tf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-l7lj2f -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster list-test-cluster-l7lj2f: failed starting cluster creation \"list-test-cluster-l7lj2f\" in resourcegroup=\"cluster-listing-zpd4tf\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-zpd4tf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-l7lj2f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00077fd60>{ msg: "failed starting cluster creation \"list-test-cluster-l7lj2f\" in resourcegroup=\"cluster-listing-zpd4tf\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-zpd4tf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-l7lj2f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011b6f30>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:14:49 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["40"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "44c2886e-d220-4d87-b3ed-5f07c57742a2", ], }, Body: <exported.nopClosingBytesReader \| 0xc00077fc80>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error: <fmt.wrapError \| 0xc00077fd80>: failed to create HCP cluster list-test-cluster-l7lj2f: failed starting cluster creation "list-test-cluster-l7lj2f" in resourcegroup="cluster-listing-zpd4tf": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-zpd4tf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-l7lj2f -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster list-test-cluster-l7lj2f: failed starting cluster creation \"list-test-cluster-l7lj2f\" in resourcegroup=\"cluster-listing-zpd4tf\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-zpd4tf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-l7lj2f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00077fd60>{ msg: "failed starting cluster creation \"list-test-cluster-l7lj2f\" in resourcegroup=\"cluster-listing-zpd4tf\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-zpd4tf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-l7lj2f\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011b6f30>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:14:49 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["40"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "44c2886e-d220-4d87-b3ed-5f07c57742a2", ], }, Body: <exported.nopClosingBytesReader \| 0xc00077fc80>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc0005b1060>: failed to create HCP cluster engineering-hcp-cluster: failed starting cluster creation "engineering-hcp-cluster" in resourcegroup="kusto-logs-j2xt9k": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-j2xt9k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed starting cluster creation \"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-j2xt9k\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-j2xt9k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005b1040>{ msg: "failed starting cluster creation \"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-j2xt9k\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-j2xt9k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012982d0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:17:57 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "7a38c7af-7c8e-43db-90f9-c06ae146cc50", ], }, Body: <exported.nopClosingBytesReader \| 0xc0005b0ba0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error: <fmt.wrapError \| 0xc0005b1060>: failed to create HCP cluster engineering-hcp-cluster: failed starting cluster creation "engineering-hcp-cluster" in resourcegroup="kusto-logs-j2xt9k": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-j2xt9k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster engineering-hcp-cluster: failed starting cluster creation \"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-j2xt9k\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-j2xt9k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005b1040>{ msg: "failed starting cluster creation \"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-j2xt9k\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-j2xt9k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012982d0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:17:57 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "7a38c7af-7c8e-43db-90f9-c06ae146cc50", ], }, Body: <exported.nopClosingBytesReader \| 0xc0005b0ba0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc000cf90a0>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed starting cluster creation "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-p2p9tv": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-p2p9tv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed starting cluster creation \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-p2p9tv\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-p2p9tv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000cf9080>{ msg: "failed starting cluster creation \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-p2p9tv\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-p2p9tv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014f8b10>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "6c8730ae-074d-4143-88b0-ae8b390dc6ab", ], "Date": [ "Mon, 16 Mar 2026 04:18:12 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["30"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000cf8fc0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 1... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error: <fmt.wrapError \| 0xc000cf90a0>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed starting cluster creation "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-p2p9tv": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-p2p9tv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-update-nodes-hcp-cluster: failed starting cluster creation \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-p2p9tv\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-p2p9tv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000cf9080>{ msg: "failed starting cluster creation \"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-p2p9tv\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-p2p9tv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014f8b10>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "6c8730ae-074d-4143-88b0-ae8b390dc6ab", ], "Date": [ "Mon, 16 Mar 2026 04:18:12 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["30"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000cf8fc0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 1... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc00012f740>: failed to create HCP cluster cluster-zstream-4-20-plc7k8: failed starting cluster creation "cluster-zstream-4-20-plc7k8" in resourcegroup="rg-zstream-upgrade-4-20-mqn7nb": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-mqn7nb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-plc7k8 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-plc7k8: failed starting cluster creation \"cluster-zstream-4-20-plc7k8\" in resourcegroup=\"rg-zstream-upgrade-4-20-mqn7nb\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-mqn7nb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-plc7k8\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00012f720>{ msg: "failed starting cluster creation \"cluster-zstream-4-20-plc7k8\" in resourcegroup=\"rg-zstream-upgrade-4-20-mqn7nb\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-mqn7nb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-plc7k8\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000a960f0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["31"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "f106c0f1-5ffe-45ed-a98f-64baa6997ef9", ], "Date": [ "Mon, 16 Mar 2026 04:18:02 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc00012ef40>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 9... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc00012f740>: failed to create HCP cluster cluster-zstream-4-20-plc7k8: failed starting cluster creation "cluster-zstream-4-20-plc7k8" in resourcegroup="rg-zstream-upgrade-4-20-mqn7nb": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-mqn7nb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-plc7k8 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-plc7k8: failed starting cluster creation \"cluster-zstream-4-20-plc7k8\" in resourcegroup=\"rg-zstream-upgrade-4-20-mqn7nb\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-mqn7nb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-plc7k8\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00012f720>{ msg: "failed starting cluster creation \"cluster-zstream-4-20-plc7k8\" in resourcegroup=\"rg-zstream-upgrade-4-20-mqn7nb\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-mqn7nb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-plc7k8\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000a960f0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["31"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "f106c0f1-5ffe-45ed-a98f-64baa6997ef9", ], "Date": [ "Mon, 16 Mar 2026 04:18:02 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc00012ef40>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 9... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc000e10980>: failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-zfrg75": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-zfrg75/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed starting cluster creation \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-zfrg75\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-zfrg75/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e10960>{ msg: "failed starting cluster creation \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-zfrg75\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-zfrg75/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000f8f350>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:16:54 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["35"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "378d7b05-d9f3-42e0-b072-96ff2fabaabb", ], }, Body: <exported.nopClosingBytesReader \| 0xc000e108a0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 3... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc000e10980>: failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-zfrg75": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-zfrg75/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed starting cluster creation \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-zfrg75\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-zfrg75/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e10960>{ msg: "failed starting cluster creation \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-zfrg75\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-zfrg75/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000f8f350>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:16:54 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["35"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "378d7b05-d9f3-42e0-b072-96ff2fabaabb", ], }, Body: <exported.nopClosingBytesReader \| 0xc000e108a0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 3... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapError \| 0xc00096cd00>: failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-fftqnz": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-fftqnz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-tags-cluster: failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-fftqnz\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-fftqnz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00096cce0>{ msg: "failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-fftqnz\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-fftqnz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001210960>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:14:59 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "559624b8-e5e1-4ac4-b328-25a6988fe090", ], }, Body: <exported.nopClosingBytesReader \| 0xc00096cc20>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 11... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapError \| 0xc00096cd00>: failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-fftqnz": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-fftqnz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-tags-cluster: failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-fftqnz\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-fftqnz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00096cce0>{ msg: "failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-fftqnz\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-fftqnz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001210960>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:14:59 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "559624b8-e5e1-4ac4-b328-25a6988fe090", ], }, Body: <exported.nopClosingBytesReader \| 0xc00096cc20>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 11... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc000b03f20>: failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-xvv4s9": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xvv4s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation \"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-xvv4s9\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xvv4s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b03f00>{ msg: "failed starting cluster creation \"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-xvv4s9\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xvv4s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ab62a0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "6fef2e8f-635d-46e9-8c86-eac613c46a08", ], "Date": [ "Mon, 16 Mar 2026 04:17:56 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["30"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], }, Body: <exported.nopClosingBytesReader \| 0xc000b03d20>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:86]: Unexpected error: <fmt.wrapError \| 0xc000b03f20>: failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-xvv4s9": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xvv4s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation \"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-xvv4s9\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xvv4s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b03f00>{ msg: "failed starting cluster creation \"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-xvv4s9\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xvv4s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ab62a0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "6fef2e8f-635d-46e9-8c86-eac613c46a08", ], "Date": [ "Mon, 16 Mar 2026 04:17:56 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["30"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], }, Body: <exported.nopClosingBytesReader \| 0xc000b03d20>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc000a89940>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-hp8mjn": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-hp8mjn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation \"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-hp8mjn\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-hp8mjn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a89920>{ msg: "failed starting cluster creation \"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-hp8mjn\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-hp8mjn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0010226c0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["29"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "72b59c9f-5470-47b7-8095-dabe5e6a5c8b", ], "Date": [ "Mon, 16 Mar 2026 04:14:50 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc000a89860>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc000a89940>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-hp8mjn": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-hp8mjn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation \"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-hp8mjn\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-hp8mjn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a89920>{ msg: "failed starting cluster creation \"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-hp8mjn\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-hp8mjn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0010226c0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["29"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "72b59c9f-5470-47b7-8095-dabe5e6a5c8b", ], "Date": [ "Mon, 16 Mar 2026 04:14:50 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc000a89860>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:99]: Unexpected error: <fmt.wrapError \| 0xc000776520>: failed starting cluster creation "admin-cred-lifecycle-qc4566" in resourcegroup="admin-credential-lifecycle-test-rs9zpk": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rs9zpk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-qc4566 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed starting cluster creation \"admin-cred-lifecycle-qc4566\" in resourcegroup=\"admin-credential-lifecycle-test-rs9zpk\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rs9zpk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-qc4566\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0015fe030>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:14:59 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["40"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "8a57294a-cd16-48d8-bcf8-086bec0a9db0", ], }, Body: <exported.nopClosingBytesReader \| 0xc0007763e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 50, 32, 101, 114, 114, 111, 114, 115, 32, 111, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 100, 101, 116, 97, 105, 108, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 101, 116, 99, 100, 46, 100, 97, 116, 97, 69, 110, 99, 114, 121, 112, 116, 105, 111, 110, 46, 99, 117, 115, 116, 111, 109, 101, 114, 77, 97, 110, 97, 103, 101, 100, 46, 107, 109, 115, 46, 118, 105, 115, 105, 98, 105, 108, 105, 116, 121, 34, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 125, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:99]: Unexpected error: <fmt.wrapError \| 0xc000776520>: failed starting cluster creation "admin-cred-lifecycle-qc4566" in resourcegroup="admin-credential-lifecycle-test-rs9zpk": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rs9zpk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-qc4566 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed starting cluster creation \"admin-cred-lifecycle-qc4566\" in resourcegroup=\"admin-credential-lifecycle-test-rs9zpk\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-rs9zpk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-qc4566\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0015fe030>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:14:59 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["40"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "8a57294a-cd16-48d8-bcf8-086bec0a9db0", ], }, Body: <exported.nopClosingBytesReader \| 0xc0007763e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 50, 32, 101, 114, 114, 111, 114, 115, 32, 111, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 100, 101, 116, 97, 105, 108, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 101, 116, 99, 100, 46, 100, 97, 116, 97, 69, 110, 99, 114, 121, 112, 116, 105, 111, 110, 46, 99, 117, 115, 116, 111, 109, 101, 114, 77, 97, 110, 97, 103, 101, 100, 46, 107, 109, 115, 46, 118, 105, 115, 105, 98, 105, 108, 105, 116, 121, 34, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 125, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error: <fmt.wrapError \| 0xc00073f740>: failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-v6hrpd": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-v6hrpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-cluster: failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-v6hrpd\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-v6hrpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00073f700>{ msg: "failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-v6hrpd\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-v6hrpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012e6b70>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:15:00 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "243ff774-2aae-4b08-a595-4aaf80f350ef", ], }, Body: <exported.nopClosingBytesReader \| 0xc00073f540>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error: <fmt.wrapError \| 0xc00073f740>: failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-v6hrpd": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-v6hrpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-cluster: failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-v6hrpd\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-v6hrpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00073f700>{ msg: "failed starting cluster creation \"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-v6hrpd\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-v6hrpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012e6b70>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:15:00 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "243ff774-2aae-4b08-a595-4aaf80f350ef", ], }, Body: <exported.nopClosingBytesReader \| 0xc00073f540>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000ea3480>: failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-pwsp6d": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-pwsp6d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed starting cluster creation \"ea-cluster\" in resourcegroup=\"external-auth-cluster-pwsp6d\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-pwsp6d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000ea3460>{ msg: "failed starting cluster creation \"ea-cluster\" in resourcegroup=\"external-auth-cluster-pwsp6d\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-pwsp6d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011faa20>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "8a12884d-371f-4a60-a346-9bde4fef1bdc", ], "Date": [ "Mon, 16 Mar 2026 04:14:58 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc000ea33a0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000ea3480>: failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-pwsp6d": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-pwsp6d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed starting cluster creation \"ea-cluster\" in resourcegroup=\"external-auth-cluster-pwsp6d\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-pwsp6d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000ea3460>{ msg: "failed starting cluster creation \"ea-cluster\" in resourcegroup=\"external-auth-cluster-pwsp6d\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-pwsp6d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011faa20>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["32"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "8a12884d-371f-4a60-a346-9bde4fef1bdc", ], "Date": [ "Mon, 16 Mar 2026 04:14:58 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc000ea33a0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: Unexpected error: <fmt.wrapError \| 0xc00060d400>: failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-x86ztc": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-x86ztc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed starting cluster creation \"basic-hcp-cluster\" in resourcegroup=\"customer-rg-x86ztc\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-x86ztc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00137d320>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "9bf7be7d-f58d-4898-9dec-9ecd01e3efa3", ], "Date": [ "Mon, 16 Mar 2026 04:15:37 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["34"], }, Body: <exported.nopClosingBytesReader \| 0xc00060d160>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 50, 32, 101, 114, 114, 111, 114, 115, 32, 111, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 100, 101, 116, 97, 105, 108, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 101, 116, 99, 100, 46, 100, 97, 116, 97, 69, 110, 99, 114, 121, 112, 116, 105, 111, 110, 46, 99, 117, 115, 116, 111, 109, 101, 114, 77, 97, 110, 97, 103, 101, 100, 46, 107, 109, 115, 46, 118, 105, 115, 105, 98, 105, 108, 105, 116, 121, 34, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 125, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 4... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: Unexpected error: <fmt.wrapError \| 0xc00060d400>: failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-x86ztc": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-x86ztc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed starting cluster creation \"basic-hcp-cluster\" in resourcegroup=\"customer-rg-x86ztc\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-x86ztc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00137d320>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "9bf7be7d-f58d-4898-9dec-9ecd01e3efa3", ], "Date": [ "Mon, 16 Mar 2026 04:15:37 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["34"], }, Body: <exported.nopClosingBytesReader \| 0xc00060d160>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 50, 32, 101, 114, 114, 111, 114, 115, 32, 111, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 100, 101, 116, 97, 105, 108, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 113, 117, 105, 114, 101, 100, 32, 118, 97, 108, 117, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 46, 101, 116, 99, 100, 46, 100, 97, 116, 97, 69, 110, 99, 114, 121, 112, 116, 105, 111, 110, 46, 99, 117, 115, 116, 111, 109, 101, 114, 77, 97, 110, 97, 103, 101, 100, 46, 107, 109, 115, 46, 118, 105, 115, 105, 98, 105, 108, 105, 116, 121, 34, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 125, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 4... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error: <fmt.wrapError \| 0xc00074e520>: failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-cldhf9": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-cldhf9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-cldhf9\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-cldhf9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00074e440>{ msg: "failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-cldhf9\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-cldhf9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001131d10>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "6cfc33f1-9c50-4fb2-9e3f-80eb42e82879", ], "Date": [ "Mon, 16 Mar 2026 04:14:49 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["36"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], }, Body: <exported.nopClosingBytesReader \| 0xc000717f60>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error: <fmt.wrapError \| 0xc00074e520>: failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-cldhf9": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-cldhf9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-cldhf9\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-cldhf9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00074e440>{ msg: "failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-cldhf9\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-cldhf9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001131d10>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "6cfc33f1-9c50-4fb2-9e3f-80eb42e82879", ], "Date": [ "Mon, 16 Mar 2026 04:14:49 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["36"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], }, Body: <exported.nopClosingBytesReader \| 0xc000717f60>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc00145d620>: failed to create HCP cluster cluster-zstream-4-21-wvf2s7: failed starting cluster creation "cluster-zstream-4-21-wvf2s7" in resourcegroup="rg-zstream-upgrade-4-21-6cn5dr": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-6cn5dr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-wvf2s7 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-wvf2s7: failed starting cluster creation \"cluster-zstream-4-21-wvf2s7\" in resourcegroup=\"rg-zstream-upgrade-4-21-6cn5dr\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-6cn5dr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-wvf2s7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00145d600>{ msg: "failed starting cluster creation \"cluster-zstream-4-21-wvf2s7\" in resourcegroup=\"rg-zstream-upgrade-4-21-6cn5dr\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-6cn5dr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-wvf2s7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012b43c0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "2a932ba1-9dfa-4fda-a922-9a363b868873", ], "Date": [ "Mon, 16 Mar 2026 04:17:56 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["30"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc00145d540>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 9... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc00145d620>: failed to create HCP cluster cluster-zstream-4-21-wvf2s7: failed starting cluster creation "cluster-zstream-4-21-wvf2s7" in resourcegroup="rg-zstream-upgrade-4-21-6cn5dr": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-6cn5dr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-wvf2s7 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-wvf2s7: failed starting cluster creation \"cluster-zstream-4-21-wvf2s7\" in resourcegroup=\"rg-zstream-upgrade-4-21-6cn5dr\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-6cn5dr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-wvf2s7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00145d600>{ msg: "failed starting cluster creation \"cluster-zstream-4-21-wvf2s7\" in resourcegroup=\"rg-zstream-upgrade-4-21-6cn5dr\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-6cn5dr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-wvf2s7\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012b43c0>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "2a932ba1-9dfa-4fda-a922-9a363b868873", ], "Date": [ "Mon, 16 Mar 2026 04:17:56 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["30"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc00145d540>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 9... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc001560a60>: failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-jzmjwm": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-jzmjwm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed starting cluster creation \"node-limit-cluster\" in resourcegroup=\"node-limit-test-jzmjwm\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-jzmjwm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001560a40>{ msg: "failed starting cluster creation \"node-limit-cluster\" in resourcegroup=\"node-limit-test-jzmjwm\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-jzmjwm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011c5290>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "58c17626-9461-4842-b638-bc79c8bd43da", ], "Date": [ "Mon, 16 Mar 2026 04:17:56 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["30"], }, Body: <exported.nopClosingBytesReader \| 0xc001560980>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 3... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc001560a60>: failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-jzmjwm": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-jzmjwm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed starting cluster creation \"node-limit-cluster\" in resourcegroup=\"node-limit-test-jzmjwm\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-jzmjwm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001560a40>{ msg: "failed starting cluster creation \"node-limit-cluster\" in resourcegroup=\"node-limit-test-jzmjwm\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-jzmjwm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011c5290>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "58c17626-9461-4842-b638-bc79c8bd43da", ], "Date": [ "Mon, 16 Mar 2026 04:17:56 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["30"], }, Body: <exported.nopClosingBytesReader \| 0xc001560980>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 3... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapError \| 0xc0004934e0>: failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-w8rs7x": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-w8rs7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-w8rs7x\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-w8rs7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0004934c0>{ msg: "failed starting cluster creation \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-w8rs7x\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-w8rs7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000cfcd50>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:14:49 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["34"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "69aaf16e-7144-4085-a11e-568c44599939", ], }, Body: <exported.nopClosingBytesReader \| 0xc000493220>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 3... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapError \| 0xc0004934e0>: failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-w8rs7x": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-w8rs7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-w8rs7x\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-w8rs7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0004934c0>{ msg: "failed starting cluster creation \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-w8rs7x\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-w8rs7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000cfcd50>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Date": [ "Mon, 16 Mar 2026 04:14:49 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["34"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "69aaf16e-7144-4085-a11e-568c44599939", ], }, Body: <exported.nopClosingBytesReader \| 0xc000493220>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 3... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Unsupported value: "": supported values: "Private; provider Microsoft.RedHatOpenShift 03:27:37 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000b103a0>: failed to create HCP cluster cluster-zstream-4-19-ccltmq: failed starting cluster creation "cluster-zstream-4-19-ccltmq" in resourcegroup="rg-zstream-upgrade-4-19-d5n6c6": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-d5n6c6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ccltmq -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-ccltmq: failed starting cluster creation \"cluster-zstream-4-19-ccltmq\" in resourcegroup=\"rg-zstream-upgrade-4-19-d5n6c6\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-d5n6c6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ccltmq\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b10380>{ msg: "failed starting cluster creation \"cluster-zstream-4-19-ccltmq\" in resourcegroup=\"rg-zstream-upgrade-4-19-d5n6c6\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-d5n6c6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ccltmq\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000251980>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "55e0ab98-2366-4911-9f7b-f4c7195b7447", ], "Date": [ "Mon, 16 Mar 2026 04:18:03 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["1035"], }, Body: <exported.nopClosingBytesReader \| 0xc000b10040>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000b103a0>: failed to create HCP cluster cluster-zstream-4-19-ccltmq: failed starting cluster creation "cluster-zstream-4-19-ccltmq" in resourcegroup="rg-zstream-upgrade-4-19-d5n6c6": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-d5n6c6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ccltmq -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "error": { "code": "MultipleErrorsOccurred", "message": "2 errors occurred", "details": [ { "code": "InvalidRequestContent", "message": "Required value", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" }, { "code": "InvalidRequestContent", "message": "Unsupported value: \"\": supported values: \"Private\", \"Public\"", "target": "properties.etcd.dataEncryption.customerManaged.kms.visibility" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-ccltmq: failed starting cluster creation \"cluster-zstream-4-19-ccltmq\" in resourcegroup=\"rg-zstream-upgrade-4-19-d5n6c6\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-d5n6c6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ccltmq\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b10380>{ msg: "failed starting cluster creation \"cluster-zstream-4-19-ccltmq\" in resourcegroup=\"rg-zstream-upgrade-4-19-d5n6c6\": PUT https://rp.j4892416.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-d5n6c6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-19-ccltmq\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"2 errors occurred\",\n \"details\": [\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Required value\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n },\n {\n \"code\": \"InvalidRequestContent\",\n \"message\": \"Unsupported value: \\\"\\\": supported values: \\\"Private\\\", \\\"Public\\\"\",\n \"target\": \"properties.etcd.dataEncryption.customerManaged.kms.visibility\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000251980>{ ErrorCode: "MultipleErrorsOccurred", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "MultipleErrorsOccurred", ], "X-Ms-Request-Id": [ "55e0ab98-2366-4911-9f7b-f4c7195b7447", ], "Date": [ "Mon, 16 Mar 2026 04:18:03 GMT", ], "Content-Length": ["614"], "X-Envoy-Upstream-Service-Time": ["1035"], }, Body: <exported.nopClosingBytesReader \| 0xc000b10040>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
03:15:42	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4d5b425	n/a	n/a	passed	#4475 (closed)	0	n/a
01:45:27	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head dfb6412	n/a	n/a	passed	#4470 (open)	0	n/a
01:21:57	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 428d780 · merge 8b7576c	n/a	unknown	failed	#4473 (merged)	1	non-artifact-backed failure (no junit artifacts)
00:53:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 54fcfdd · merge 205984e	n/a	n/a	passed	#4449 (merged)	0	n/a
00:35:56	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e4f48ae · merge e4f48ae	n/a	n/a	passed	#4471 (merged)	0	n/a