CIHealth Run Log

Environment: PROD

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
09:38:45	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	8	e2e (8) ERROR CODE: DeploymentFailed; detail code InternalServerError 09:38:45 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:89]: Unexpected error: <fmt.wrapError \| 0xc0016988c0>: failed to create customer-infra: failed waiting for deployment "customer-infra-cidr-connectivity-test-2n2dpm" in resourcegroup="e2e-cidr-connectivity-jr9jc2" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-jr9jc2/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-2n2dpm/operationStatuses/08584276071663690510 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094222Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a468c4a2-9edb-4ef5-bf9f-734be2ee8178', request correlation id '9089fd1b-9ad7-48ea-9e52-4feecb5800da'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cidr-connectivity-test-2n2dpm\" in resourcegroup=\"e2e-cidr-connectivity-jr9jc2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-jr9jc2/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-2n2dpm/operationStatuses/08584276071663690510\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094222Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a468c4a2-9edb-4ef5-bf9f-734be2ee8178', request correlation id '9089fd1b-9ad7-48ea-9e52-4feecb5800da'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0016988a0>{ msg: "failed waiting for deployment \"customer-infra-cidr-connectivity-test-2n2dpm\" in resourcegroup=\"e2e-cidr-connectivity-jr9jc2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-jr9jc2/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-2n2dpm/operationStatuses/08584276071663690510\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094222Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a468c4a2-9edb-4ef5-bf9f-734be2ee8178', request correlation id '9089fd1b-9ad7-48ea-9e52-4feecb5800da'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0009a2bd0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["655"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Content-Type-Options": ["nosniff"], "Date": [ "Fri, 20 Mar 2026 09:42:31 GMT", ], "Pragma": ["no-cache"], "X-Ms-Routing-Request-Id": [ "CENTRALINDIA:20260... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:89]: Unexpected error: <fmt.wrapError \| 0xc0016988c0>: failed to create customer-infra: failed waiting for deployment "customer-infra-cidr-connectivity-test-2n2dpm" in resourcegroup="e2e-cidr-connectivity-jr9jc2" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-jr9jc2/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-2n2dpm/operationStatuses/08584276071663690510 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094222Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a468c4a2-9edb-4ef5-bf9f-734be2ee8178', request correlation id '9089fd1b-9ad7-48ea-9e52-4feecb5800da'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cidr-connectivity-test-2n2dpm\" in resourcegroup=\"e2e-cidr-connectivity-jr9jc2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-jr9jc2/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-2n2dpm/operationStatuses/08584276071663690510\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094222Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a468c4a2-9edb-4ef5-bf9f-734be2ee8178', request correlation id '9089fd1b-9ad7-48ea-9e52-4feecb5800da'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0016988a0>{ msg: "failed waiting for deployment \"customer-infra-cidr-connectivity-test-2n2dpm\" in resourcegroup=\"e2e-cidr-connectivity-jr9jc2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-jr9jc2/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-2n2dpm/operationStatuses/08584276071663690510\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094222Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a468c4a2-9edb-4ef5-bf9f-734be2ee8178', request correlation id '9089fd1b-9ad7-48ea-9e52-4feecb5800da'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0009a2bd0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["655"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Content-Type-Options": ["nosniff"], "Date": [ "Fri, 20 Mar 2026 09:42:31 GMT", ], "Pragma": ["no-cache"], "X-Ms-Routing-Request-Id": [ "CENTRALINDIA:20260... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 09:38:45 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:80]: Unexpected error: <fmt.wrapError \| 0xc000a2bd60>: failed to create customer-infra: failed waiting for deployment "customer-infra-arm64-vm-hcp-cluster-7sdmld" in resourcegroup="arm64-vm-cluster-bhd6ww" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-arm64-vm-hcp-cluster-7sdmld\" in resourcegroup=\"arm64-vm-cluster-bhd6ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a2bd40>{ msg: "failed waiting for deployment \"customer-infra-arm64-vm-hcp-cluster-7sdmld\" in resourcegroup=\"arm64-vm-cluster-bhd6ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0008036e0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Correlation-Request-Id": [ "654f520d-e08b-4b13-9d40-7492126aea91", ], "X-Content-Type-Options": ["nosniff"], "Expires": ["-1"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:80]: Unexpected error: <fmt.wrapError \| 0xc000a2bd60>: failed to create customer-infra: failed waiting for deployment "customer-infra-arm64-vm-hcp-cluster-7sdmld" in resourcegroup="arm64-vm-cluster-bhd6ww" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-arm64-vm-hcp-cluster-7sdmld\" in resourcegroup=\"arm64-vm-cluster-bhd6ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a2bd40>{ msg: "failed waiting for deployment \"customer-infra-arm64-vm-hcp-cluster-7sdmld\" in resourcegroup=\"arm64-vm-cluster-bhd6ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/arm64-vm-cluster-bhd6ww/providers/Microsoft.Resources/deployments/customer-infra-arm64-vm-hcp-cluster-7sdmld/operationStatuses/08584276071679720129\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094216Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '9d0d4b1f-6cd4-46ef-9360-db352e410b43', request correlation id '086357ea-a5df-43aa-9ade-32430cd392a8'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0008036e0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Correlation-Request-Id": [ "654f520d-e08b-4b13-9d40-7492126aea91", ], "X-Content-Type-Options": ["nosniff"], "Expires": ["-1"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 09:38:45 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc0011ceec0>: failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-zstream-4-20-w5kd42-sclm5g" in resourcegroup="rg-zstream-upgrade-4-20-mjtqvq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-zstream-upgrade-4-20-mjtqvq/providers/Microsoft.Resources/deployments/customer-infra-cluster-zstream-4-20-w5kd42-sclm5g/operationStatuses/08584276069265573537 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094634Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '82c3deed-9646-4c27-99e2-130719af36ca', request correlation id 'efefbec6-8c68-4fe7-84e9-3b16a241a5b4'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-zstream-4-20-w5kd42-sclm5g\" in resourcegroup=\"rg-zstream-upgrade-4-20-mjtqvq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-zstream-upgrade-4-20-mjtqvq/providers/Microsoft.Resources/deployments/customer-infra-cluster-zstream-4-20-w5kd42-sclm5g/operationStatuses/08584276069265573537\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094634Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '82c3deed-9646-4c27-99e2-130719af36ca', request correlation id 'efefbec6-8c68-4fe7-84e9-3b16a241a5b4'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0011ceea0>{ msg: "failed waiting for deployment \"customer-infra-cluster-zstream-4-20-w5kd42-sclm5g\" in resourcegroup=\"rg-zstream-upgrade-4-20-mjtqvq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-zstream-upgrade-4-20-mjtqvq/providers/Microsoft.Resources/deployments/customer-infra-cluster-zstream-4-20-w5kd42-sclm5g/operationStatuses/08584276069265573537\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094634Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '82c3deed-9646-4c27-99e2-130719af36ca', request correlation id 'efefbec6-8c68-4fe7-84e9-3b16a241a5b4'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0009b5530>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Msedge-Ref": [ "Ref A: C32E0FBEF19E41938E8F8432FEF57C67 Ref B: CH1AA2020617019 Ref C: 2026-03-20T09:46:41Z", ], "Pragma": ["no-cache"], "Content-Length": ["655"], "Content-Type": [ "application/jso... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc0011ceec0>: failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-zstream-4-20-w5kd42-sclm5g" in resourcegroup="rg-zstream-upgrade-4-20-mjtqvq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-zstream-upgrade-4-20-mjtqvq/providers/Microsoft.Resources/deployments/customer-infra-cluster-zstream-4-20-w5kd42-sclm5g/operationStatuses/08584276069265573537 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094634Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '82c3deed-9646-4c27-99e2-130719af36ca', request correlation id 'efefbec6-8c68-4fe7-84e9-3b16a241a5b4'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-zstream-4-20-w5kd42-sclm5g\" in resourcegroup=\"rg-zstream-upgrade-4-20-mjtqvq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-zstream-upgrade-4-20-mjtqvq/providers/Microsoft.Resources/deployments/customer-infra-cluster-zstream-4-20-w5kd42-sclm5g/operationStatuses/08584276069265573537\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094634Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '82c3deed-9646-4c27-99e2-130719af36ca', request correlation id 'efefbec6-8c68-4fe7-84e9-3b16a241a5b4'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0011ceea0>{ msg: "failed waiting for deployment \"customer-infra-cluster-zstream-4-20-w5kd42-sclm5g\" in resourcegroup=\"rg-zstream-upgrade-4-20-mjtqvq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-zstream-upgrade-4-20-mjtqvq/providers/Microsoft.Resources/deployments/customer-infra-cluster-zstream-4-20-w5kd42-sclm5g/operationStatuses/08584276069265573537\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094634Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '82c3deed-9646-4c27-99e2-130719af36ca', request correlation id 'efefbec6-8c68-4fe7-84e9-3b16a241a5b4'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0009b5530>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Msedge-Ref": [ "Ref A: C32E0FBEF19E41938E8F8432FEF57C67 Ref B: CH1AA2020617019 Ref C: 2026-03-20T09:46:41Z", ], "Pragma": ["no-cache"], "Content-Length": ["655"], "Content-Type": [ "application/jso... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 09:38:45 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:81]: Unexpected error: <fmt.wrapError \| 0xc00095d240>: failed to create customer-infra: failed waiting for deployment "customer-infra-negative-tests-cluster-ntc864" in resourcegroup="rg-negative-tests-bnmj86" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-negative-tests-bnmj86/providers/Microsoft.Resources/deployments/customer-infra-negative-tests-cluster-ntc864/operationStatuses/08584276069419727049 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094609Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'cb698f35-dbf2-472f-85d8-46e266307146', request correlation id '1dc775d6-a615-4d97-8b2f-fc3571abd0a4'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-negative-tests-cluster-ntc864\" in resourcegroup=\"rg-negative-tests-bnmj86\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-negative-tests-bnmj86/providers/Microsoft.Resources/deployments/customer-infra-negative-tests-cluster-ntc864/operationStatuses/08584276069419727049\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094609Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'cb698f35-dbf2-472f-85d8-46e266307146', request correlation id '1dc775d6-a615-4d97-8b2f-fc3571abd0a4'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00095d220>{ msg: "failed waiting for deployment \"customer-infra-negative-tests-cluster-ntc864\" in resourcegroup=\"rg-negative-tests-bnmj86\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-negative-tests-bnmj86/providers/Microsoft.Resources/deployments/customer-infra-negative-tests-cluster-ntc864/operationStatuses/08584276069419727049\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094609Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'cb698f35-dbf2-472f-85d8-46e266307146', request correlation id '1dc775d6-a615-4d97-8b2f-fc3571abd0a4'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000db29c0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Routing-Request-Id": [ "CENTRALINDIA:20260320T094615Z:743da700-4576-41d5-a9ad-f747a272ed30", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Length": ["655"], "Content-Type": [ "application/json; charset=utf-8", ],... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:81]: Unexpected error: <fmt.wrapError \| 0xc00095d240>: failed to create customer-infra: failed waiting for deployment "customer-infra-negative-tests-cluster-ntc864" in resourcegroup="rg-negative-tests-bnmj86" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-negative-tests-bnmj86/providers/Microsoft.Resources/deployments/customer-infra-negative-tests-cluster-ntc864/operationStatuses/08584276069419727049 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094609Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'cb698f35-dbf2-472f-85d8-46e266307146', request correlation id '1dc775d6-a615-4d97-8b2f-fc3571abd0a4'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-negative-tests-cluster-ntc864\" in resourcegroup=\"rg-negative-tests-bnmj86\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-negative-tests-bnmj86/providers/Microsoft.Resources/deployments/customer-infra-negative-tests-cluster-ntc864/operationStatuses/08584276069419727049\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094609Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'cb698f35-dbf2-472f-85d8-46e266307146', request correlation id '1dc775d6-a615-4d97-8b2f-fc3571abd0a4'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00095d220>{ msg: "failed waiting for deployment \"customer-infra-negative-tests-cluster-ntc864\" in resourcegroup=\"rg-negative-tests-bnmj86\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-negative-tests-bnmj86/providers/Microsoft.Resources/deployments/customer-infra-negative-tests-cluster-ntc864/operationStatuses/08584276069419727049\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094609Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'cb698f35-dbf2-472f-85d8-46e266307146', request correlation id '1dc775d6-a615-4d97-8b2f-fc3571abd0a4'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000db29c0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Routing-Request-Id": [ "CENTRALINDIA:20260320T094615Z:743da700-4576-41d5-a9ad-f747a272ed30", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Length": ["655"], "Content-Type": [ "application/json; charset=utf-8", ],... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 09:38:45 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:88]: Unexpected error: <fmt.wrapError \| 0xc000a2f0a0>: failed to create customer-infra: failed waiting for deployment "customer-infra-autoscaling-hcp-cluster-xmkl48" in resourcegroup="autoscaling-cluster-54vlxn" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/autoscaling-cluster-54vlxn/providers/Microsoft.Resources/deployments/customer-infra-autoscaling-hcp-cluster-xmkl48/operationStatuses/08584276071662655044 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094235Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'd879ca32-ec1f-4f1d-9b61-914bc7f3aa52', request correlation id '24e5769c-113d-49e3-a2fe-a79a4ce994bb'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-autoscaling-hcp-cluster-xmkl48\" in resourcegroup=\"autoscaling-cluster-54vlxn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/autoscaling-cluster-54vlxn/providers/Microsoft.Resources/deployments/customer-infra-autoscaling-hcp-cluster-xmkl48/operationStatuses/08584276071662655044\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094235Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'd879ca32-ec1f-4f1d-9b61-914bc7f3aa52', request correlation id '24e5769c-113d-49e3-a2fe-a79a4ce994bb'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a2f080>{ msg: "failed waiting for deployment \"customer-infra-autoscaling-hcp-cluster-xmkl48\" in resourcegroup=\"autoscaling-cluster-54vlxn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/autoscaling-cluster-54vlxn/providers/Microsoft.Resources/deployments/customer-infra-autoscaling-hcp-cluster-xmkl48/operationStatuses/08584276071662655044\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094235Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'd879ca32-ec1f-4f1d-9b61-914bc7f3aa52', request correlation id '24e5769c-113d-49e3-a2fe-a79a4ce994bb'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000b61ef0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["655"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Msedge-Ref": [ "Ref A: B86F241788DB4184AB9A686C0E97EEC8 Ref B: CH1AA2020620025 Ref C: 2026-03-20T09:42:42Z", ], "Pragma": ["no-cache"], "Content-Type": [ "application/json; charset=utf-8", ], "X-Ms... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:88]: Unexpected error: <fmt.wrapError \| 0xc000a2f0a0>: failed to create customer-infra: failed waiting for deployment "customer-infra-autoscaling-hcp-cluster-xmkl48" in resourcegroup="autoscaling-cluster-54vlxn" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/autoscaling-cluster-54vlxn/providers/Microsoft.Resources/deployments/customer-infra-autoscaling-hcp-cluster-xmkl48/operationStatuses/08584276071662655044 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094235Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'd879ca32-ec1f-4f1d-9b61-914bc7f3aa52', request correlation id '24e5769c-113d-49e3-a2fe-a79a4ce994bb'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-autoscaling-hcp-cluster-xmkl48\" in resourcegroup=\"autoscaling-cluster-54vlxn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/autoscaling-cluster-54vlxn/providers/Microsoft.Resources/deployments/customer-infra-autoscaling-hcp-cluster-xmkl48/operationStatuses/08584276071662655044\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094235Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'd879ca32-ec1f-4f1d-9b61-914bc7f3aa52', request correlation id '24e5769c-113d-49e3-a2fe-a79a4ce994bb'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a2f080>{ msg: "failed waiting for deployment \"customer-infra-autoscaling-hcp-cluster-xmkl48\" in resourcegroup=\"autoscaling-cluster-54vlxn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/autoscaling-cluster-54vlxn/providers/Microsoft.Resources/deployments/customer-infra-autoscaling-hcp-cluster-xmkl48/operationStatuses/08584276071662655044\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094235Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'd879ca32-ec1f-4f1d-9b61-914bc7f3aa52', request correlation id '24e5769c-113d-49e3-a2fe-a79a4ce994bb'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000b61ef0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["655"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Msedge-Ref": [ "Ref A: B86F241788DB4184AB9A686C0E97EEC8 Ref B: CH1AA2020620025 Ref C: 2026-03-20T09:42:42Z", ], "Pragma": ["no-cache"], "Content-Type": [ "application/json; charset=utf-8", ], "X-Ms... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 09:38:45 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000f82360>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-6fd9wv-x2gpnw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f82340>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f82320>{ msg: "failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:85]: Unexpected error: <fmt.wrapError \| 0xc000f82360>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-6fd9wv-x2gpnw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f82340>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094914Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f9637488-a80a-49b4-9d85-d6a0f597d8f4', request correlation id 'c6f7a3ea-7476-48bb-9fbd-25ee8524741c'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f82320>{ msg: "failed waiting for deployment \"mi-cluster-zstream-4-19-6fd9wv-x2gpnw\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-6fd9wv-x2gpnw/operationStatuses/08584276067733222695\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 09:38:45 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error: <fmt.wrapError \| 0xc000498b00>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-cluster-wbpcld" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-wbpcld/operationStatuses/08584276071330702004 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094259Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '938b2578-7a35-4407-b3d7-9bc1d1d9beb6', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\"\r\n }\r\n}" }, { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094256Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b0fb71ff-8ade-4169-8256-c647afd4c712', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-cluster-wbpcld\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-wbpcld/operationStatuses/08584276071330702004\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094259Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '938b2578-7a35-4407-b3d7-9bc1d1d9beb6', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094256Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b0fb71ff-8ade-4169-8256-c647afd4c712', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000498ae0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-cluster-wbpcld\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-wbpcld/operationStatuses/08584276071330702004\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094259Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '938b2578-7a35-4407-b3d7-9bc1d1d9beb6', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094256Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b0fb71ff-8ade-4169-8256-c647afd4c712', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000498aa0>{ msg: "failed waiting for depl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error: <fmt.wrapError \| 0xc000498b00>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-cluster-wbpcld" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-wbpcld/operationStatuses/08584276071330702004 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094259Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '938b2578-7a35-4407-b3d7-9bc1d1d9beb6', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\"\r\n }\r\n}" }, { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094256Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b0fb71ff-8ade-4169-8256-c647afd4c712', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-cluster-wbpcld\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-wbpcld/operationStatuses/08584276071330702004\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094259Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '938b2578-7a35-4407-b3d7-9bc1d1d9beb6', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094256Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b0fb71ff-8ade-4169-8256-c647afd4c712', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000498ae0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-basic-cluster-wbpcld\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-wbpcld/operationStatuses/08584276071330702004\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094259Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '938b2578-7a35-4407-b3d7-9bc1d1d9beb6', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094256Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b0fb71ff-8ade-4169-8256-c647afd4c712', request correlation id '4e3a5e4e-4924-422c-848a-7ebf070d4c86'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000498aa0>{ msg: "failed waiting for depl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 09:38:45 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:81]: Unexpected error: <fmt.wrapError \| 0xc000bc5580>: failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-t59jtk" in resourcegroup="rg-np-labels-taints-gt77vc" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-np-labels-taints-t59jtk\" in resourcegroup=\"rg-np-labels-taints-gt77vc\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000bc5560>{ msg: "failed waiting for deployment \"customer-infra-cluster-np-labels-taints-t59jtk\" in resourcegroup=\"rg-np-labels-taints-gt77vc\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011eaea0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Request-Id": [ "af79b60e-5587-4117-9e37-523f09825369", ], "Date": [ "Fri, 20 Mar 2026 09:44:49 GMT", ], "Content-Type": [ "application/json; charset=utf-8", ], "Expires": ["-1"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:81]: Unexpected error: <fmt.wrapError \| 0xc000bc5580>: failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-t59jtk" in resourcegroup="rg-np-labels-taints-gt77vc" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-np-labels-taints-t59jtk\" in resourcegroup=\"rg-np-labels-taints-gt77vc\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000bc5560>{ msg: "failed waiting for deployment \"customer-infra-cluster-np-labels-taints-t59jtk\" in resourcegroup=\"rg-np-labels-taints-gt77vc\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-gt77vc/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-t59jtk/operationStatuses/08584276070386670482\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T094442Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7da13b4c-2074-4ed2-8674-1b1ea2937fdc', request correlation id '4075b009-7643-490d-b40a-fd5cab0af989'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011eaea0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Request-Id": [ "af79b60e-5587-4117-9e37-523f09825369", ], "Date": [ "Fri, 20 Mar 2026 09:44:49 GMT", ], "Content-Type": [ "application/json; charset=utf-8", ], "Expires": ["-1"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
08:26:40	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	5	e2e (5) ERROR CODE: DeploymentFailed; detail code InternalServerError 08:26:40 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:85]: Unexpected error: <fmt.wrapError \| 0xc0005db5c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-tls-endpoint-hcp-cluster-vs4ds4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-vs4ds4/operationStatuses/08584276114890211861 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083045Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '3470052b-9901-4258-b1ec-c30663cc2b31', request correlation id 'de6e37c3-2b54-4fb7-9e27-9e5a4f06b724'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-tls-endpoint-hcp-cluster-vs4ds4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-vs4ds4/operationStatuses/08584276114890211861\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083045Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '3470052b-9901-4258-b1ec-c30663cc2b31', request correlation id 'de6e37c3-2b54-4fb7-9e27-9e5a4f06b724'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005db580>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-tls-endpoint-hcp-cluster-vs4ds4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-vs4ds4/operationStatuses/08584276114890211861\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083045Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '3470052b-9901-4258-b1ec-c30663cc2b31', request correlation id 'de6e37c3-2b54-4fb7-9e27-9e5a4f06b724'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005db560>{ msg: "failed waiting for deployment \"mi-tls-endpoint-hcp-cluster-vs4ds4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-vs4ds4/operationStatuses/08584276114890211861\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"Re... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:85]: Unexpected error: <fmt.wrapError \| 0xc0005db5c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-tls-endpoint-hcp-cluster-vs4ds4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-vs4ds4/operationStatuses/08584276114890211861 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083045Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '3470052b-9901-4258-b1ec-c30663cc2b31', request correlation id 'de6e37c3-2b54-4fb7-9e27-9e5a4f06b724'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-tls-endpoint-hcp-cluster-vs4ds4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-vs4ds4/operationStatuses/08584276114890211861\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083045Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '3470052b-9901-4258-b1ec-c30663cc2b31', request correlation id 'de6e37c3-2b54-4fb7-9e27-9e5a4f06b724'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005db580>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-tls-endpoint-hcp-cluster-vs4ds4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-vs4ds4/operationStatuses/08584276114890211861\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083045Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '3470052b-9901-4258-b1ec-c30663cc2b31', request correlation id 'de6e37c3-2b54-4fb7-9e27-9e5a4f06b724'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005db560>{ msg: "failed waiting for deployment \"mi-tls-endpoint-hcp-cluster-vs4ds4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-vs4ds4/operationStatuses/08584276114890211861\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"Re... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 08:26:40 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:75]: Unexpected error: <fmt.wrapError \| 0xc0009013c0>: failed to create customer-infra: failed waiting for deployment "customer-infra-ea-listhgnbmt-lp5wsg" in resourcegroup="ea-list-mbbgvj" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/ea-list-mbbgvj/providers/Microsoft.Resources/deployments/customer-infra-ea-listhgnbmt-lp5wsg/operationStatuses/08584276112221915280 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083443Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'dd2cb401-029a-41b8-ad66-5eebd73844c0', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\"\r\n }\r\n}" }, { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083438Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '970c6a21-15a4-42af-ac71-76b38f20ba8f', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-ea-listhgnbmt-lp5wsg\" in resourcegroup=\"ea-list-mbbgvj\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/ea-list-mbbgvj/providers/Microsoft.Resources/deployments/customer-infra-ea-listhgnbmt-lp5wsg/operationStatuses/08584276112221915280\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083443Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'dd2cb401-029a-41b8-ad66-5eebd73844c0', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083438Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '970c6a21-15a4-42af-ac71-76b38f20ba8f', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0009013a0>{ msg: "failed waiting for deployment \"customer-infra-ea-listhgnbmt-lp5wsg\" in resourcegroup=\"ea-list-mbbgvj\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/ea-list-mbbgvj/providers/Microsoft.Resources/deployments/customer-infra-ea-listhgnbmt-lp5wsg/operationStatuses/08584276112221915280\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083443Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'dd2cb401-029a-41b8-ad66-5eebd73844c0', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083438Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '970c6a21-15a4-42af-ac71-76b38f20ba8f', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseE... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:75]: Unexpected error: <fmt.wrapError \| 0xc0009013c0>: failed to create customer-infra: failed waiting for deployment "customer-infra-ea-listhgnbmt-lp5wsg" in resourcegroup="ea-list-mbbgvj" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/ea-list-mbbgvj/providers/Microsoft.Resources/deployments/customer-infra-ea-listhgnbmt-lp5wsg/operationStatuses/08584276112221915280 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083443Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'dd2cb401-029a-41b8-ad66-5eebd73844c0', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\"\r\n }\r\n}" }, { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083438Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '970c6a21-15a4-42af-ac71-76b38f20ba8f', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-ea-listhgnbmt-lp5wsg\" in resourcegroup=\"ea-list-mbbgvj\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/ea-list-mbbgvj/providers/Microsoft.Resources/deployments/customer-infra-ea-listhgnbmt-lp5wsg/operationStatuses/08584276112221915280\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083443Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'dd2cb401-029a-41b8-ad66-5eebd73844c0', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083438Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '970c6a21-15a4-42af-ac71-76b38f20ba8f', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0009013a0>{ msg: "failed waiting for deployment \"customer-infra-ea-listhgnbmt-lp5wsg\" in resourcegroup=\"ea-list-mbbgvj\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/ea-list-mbbgvj/providers/Microsoft.Resources/deployments/customer-infra-ea-listhgnbmt-lp5wsg/operationStatuses/08584276112221915280\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083443Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'dd2cb401-029a-41b8-ad66-5eebd73844c0', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083438Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '970c6a21-15a4-42af-ac71-76b38f20ba8f', request correlation id '7e2189f7-bd6a-4ff4-843a-7ed32cf2326f'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseE... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 08:26:40 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc00100a300>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-lqlnsk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-lqlnsk/operationStatuses/08584276113123615830 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083320Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e95286b3-b9e3-4c91-bb33-129e4770621d', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\"\r\n }\r\n}" }, { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083318Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '500e4dbb-80c7-4e25-bb66-2cbb88dfb63f', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-lqlnsk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-lqlnsk/operationStatuses/08584276113123615830\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083320Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e95286b3-b9e3-4c91-bb33-129e4770621d', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083318Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '500e4dbb-80c7-4e25-bb66-2cbb88dfb63f', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00100a2e0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-lqlnsk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-lqlnsk/operationStatuses/08584276113123615830\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083320Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e95286b3-b9e3-4c91-bb33-129e4770621d', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083318Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '500e4dbb-80c7-4e25-bb66-2cbb88dfb63f', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00100a2c0>{ msg: "fai... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc00100a300>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-lqlnsk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-lqlnsk/operationStatuses/08584276113123615830 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083320Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e95286b3-b9e3-4c91-bb33-129e4770621d', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\"\r\n }\r\n}" }, { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083318Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '500e4dbb-80c7-4e25-bb66-2cbb88dfb63f', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-lqlnsk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-lqlnsk/operationStatuses/08584276113123615830\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083320Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e95286b3-b9e3-4c91-bb33-129e4770621d', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083318Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '500e4dbb-80c7-4e25-bb66-2cbb88dfb63f', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00100a2e0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-lqlnsk\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-lqlnsk/operationStatuses/08584276113123615830\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083320Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e95286b3-b9e3-4c91-bb33-129e4770621d', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083318Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '500e4dbb-80c7-4e25-bb66-2cbb88dfb63f', request correlation id '32ef014d-22c0-466e-a2e8-27bce60fe8c2'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00100a2c0>{ msg: "fai... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 08:26:40 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: Unexpected error: <fmt.wrapError \| 0xc00070b5e0>: failed to create customer-infra: failed waiting for deployment "customer-infra-ea-cluster-cwmtxg" in resourcegroup="external-auth-cluster-rppmwz" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rppmwz/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-cwmtxg/operationStatuses/08584276113429441044 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083241Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '646f2563-25ab-4a22-83e6-854c6b7d9112', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\"\r\n }\r\n}" }, { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083238Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e6f8324e-ba3f-472a-bf42-4fc393800750', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-ea-cluster-cwmtxg\" in resourcegroup=\"external-auth-cluster-rppmwz\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rppmwz/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-cwmtxg/operationStatuses/08584276113429441044\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083241Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '646f2563-25ab-4a22-83e6-854c6b7d9112', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083238Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e6f8324e-ba3f-472a-bf42-4fc393800750', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00070b5c0>{ msg: "failed waiting for deployment \"customer-infra-ea-cluster-cwmtxg\" in resourcegroup=\"external-auth-cluster-rppmwz\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rppmwz/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-cwmtxg/operationStatuses/08584276113429441044\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083241Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '646f2563-25ab-4a22-83e6-854c6b7d9112', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083238Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e6f8324e-ba3f-472a-bf42-4fc393800750', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n------------------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: Unexpected error: <fmt.wrapError \| 0xc00070b5e0>: failed to create customer-infra: failed waiting for deployment "customer-infra-ea-cluster-cwmtxg" in resourcegroup="external-auth-cluster-rppmwz" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rppmwz/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-cwmtxg/operationStatuses/08584276113429441044 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083241Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '646f2563-25ab-4a22-83e6-854c6b7d9112', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\"\r\n }\r\n}" }, { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083238Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e6f8324e-ba3f-472a-bf42-4fc393800750', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-ea-cluster-cwmtxg\" in resourcegroup=\"external-auth-cluster-rppmwz\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rppmwz/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-cwmtxg/operationStatuses/08584276113429441044\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083241Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '646f2563-25ab-4a22-83e6-854c6b7d9112', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083238Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e6f8324e-ba3f-472a-bf42-4fc393800750', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00070b5c0>{ msg: "failed waiting for deployment \"customer-infra-ea-cluster-cwmtxg\" in resourcegroup=\"external-auth-cluster-rppmwz\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rppmwz/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-cwmtxg/operationStatuses/08584276113429441044\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083241Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '646f2563-25ab-4a22-83e6-854c6b7d9112', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083238Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e6f8324e-ba3f-472a-bf42-4fc393800750', request correlation id '3cd16e60-c93b-4492-9a85-ba52251d353d'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n------------------------------------------------------------------------------... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.ManagedIdentity 08:26:40 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error: <fmt.wrapError \| 0xc0016053a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-9d7jkm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-9d7jkm/operationStatuses/08584276114880859088 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-jg7phn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083028Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a7739b02-f190-492e-b6ca-5904c97135c6', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0ec4bb4f-3c7a-4a85-9ef0-62497151a6f0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5fcbd0fd-49b3-4339-b567-de1c1596aaa0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '92858e8c-aa82-4d12-af4e-bbaf740beedc', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083026Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5aedb7a4-fd70-4e59-8a0f-c36fcb99fe4a', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083026Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '6ad7864e-528a-4bd7-a91c-c7eead01227a', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083025Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b474b493-2d00-4cd4-8b16-04c1849437da', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083024Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e178a66f-7a87-4e67-8562-6aafb6036181', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083023Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e59eefa1-b6f0-4d97-8d61-9506dc6c5ff4', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-9d7jkm\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-9d7jkm/operationStatuses/08584276114880859088\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-jg7phn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083028Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a7739b02-f190-492e-b6ca-5904c97135c6', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0ec4bb4f-3c7a-4a85-9ef0-62497151a6f0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5fcbd0fd-49b3-4339-b567-de1c1596aaa0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\\",\\r\\n \\\"message\\\": \\\"En... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error: <fmt.wrapError \| 0xc0016053a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-9d7jkm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-9d7jkm/operationStatuses/08584276114880859088 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-jg7phn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083028Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a7739b02-f190-492e-b6ca-5904c97135c6', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0ec4bb4f-3c7a-4a85-9ef0-62497151a6f0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5fcbd0fd-49b3-4339-b567-de1c1596aaa0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '92858e8c-aa82-4d12-af4e-bbaf740beedc', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083026Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5aedb7a4-fd70-4e59-8a0f-c36fcb99fe4a', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083026Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '6ad7864e-528a-4bd7-a91c-c7eead01227a', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083025Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b474b493-2d00-4cd4-8b16-04c1849437da', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083024Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e178a66f-7a87-4e67-8562-6aafb6036181', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n },\r\n {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/kms\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T083023Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e59eefa1-b6f0-4d97-8d61-9506dc6c5ff4', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-basic-hcp-cluster-9d7jkm\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-9d7jkm/operationStatuses/08584276114880859088\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-jg7phn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083028Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a7739b02-f190-492e-b6ca-5904c97135c6', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-file-csi-driver\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0ec4bb4f-3c7a-4a85-9ef0-62497151a6f0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T083027Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '5fcbd0fd-49b3-4339-b567-de1c1596aaa0', request correlation id '6c19fc94-efec-4cab-8d4f-81ceaf7a1471'.\\\"\\r\\n },\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-107/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\\\",\\r\\n \\\"message\\\": \\\"En... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
06:48:51	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	5	e2e (5) ERROR CODE: DeploymentFailed; detail code InternalServerError 06:48:51 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:81]: Unexpected error: <fmt.wrapError \| 0xc000b113c0>: failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-qf4qn7" in resourcegroup="rg-np-labels-taints-pwgws7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-pwgws7/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-qf4qn7/operationStatuses/08584276167094579006 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T070316Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e79b13b8-9887-41ca-84fb-d10db148c964', request correlation id '45b50caa-ef21-4082-9126-15814401c225'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-np-labels-taints-qf4qn7\" in resourcegroup=\"rg-np-labels-taints-pwgws7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-pwgws7/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-qf4qn7/operationStatuses/08584276167094579006\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070316Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e79b13b8-9887-41ca-84fb-d10db148c964', request correlation id '45b50caa-ef21-4082-9126-15814401c225'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b11340>{ msg: "failed waiting for deployment \"customer-infra-cluster-np-labels-taints-qf4qn7\" in resourcegroup=\"rg-np-labels-taints-pwgws7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-pwgws7/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-qf4qn7/operationStatuses/08584276167094579006\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070316Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e79b13b8-9887-41ca-84fb-d10db148c964', request correlation id '45b50caa-ef21-4082-9126-15814401c225'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ed53b0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Fri, 20 Mar 2026 07:03:27 GMT", ], "Content-Length": ["655"], "X-Ms-Correlation-Request-Id": [ "a9d5e1b3-5934-4646-a7a5-dee483a936b2", ], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "X-Ms... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:81]: Unexpected error: <fmt.wrapError \| 0xc000b113c0>: failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-qf4qn7" in resourcegroup="rg-np-labels-taints-pwgws7" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-pwgws7/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-qf4qn7/operationStatuses/08584276167094579006 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T070316Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e79b13b8-9887-41ca-84fb-d10db148c964', request correlation id '45b50caa-ef21-4082-9126-15814401c225'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-np-labels-taints-qf4qn7\" in resourcegroup=\"rg-np-labels-taints-pwgws7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-pwgws7/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-qf4qn7/operationStatuses/08584276167094579006\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070316Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e79b13b8-9887-41ca-84fb-d10db148c964', request correlation id '45b50caa-ef21-4082-9126-15814401c225'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b11340>{ msg: "failed waiting for deployment \"customer-infra-cluster-np-labels-taints-qf4qn7\" in resourcegroup=\"rg-np-labels-taints-pwgws7\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-pwgws7/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-qf4qn7/operationStatuses/08584276167094579006\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070316Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'e79b13b8-9887-41ca-84fb-d10db148c964', request correlation id '45b50caa-ef21-4082-9126-15814401c225'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ed53b0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Fri, 20 Mar 2026 07:03:27 GMT", ], "Content-Length": ["655"], "X-Ms-Correlation-Request-Id": [ "a9d5e1b3-5934-4646-a7a5-dee483a936b2", ], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "X-Ms... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 06:48:51 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc0013c7720>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-gm44fm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-gm44fm/operationStatuses/08584276167533741052 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T070251Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '44bd836b-527a-43f9-bbba-dd2564a97a3c', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\"\r\n }\r\n}" }, { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T070252Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0e93ac46-2e42-40bd-a4a6-ef9e55d0dd13', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-gm44fm\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-gm44fm/operationStatuses/08584276167533741052\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070251Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '44bd836b-527a-43f9-bbba-dd2564a97a3c', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070252Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0e93ac46-2e42-40bd-a4a6-ef9e55d0dd13', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0013c7700>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-gm44fm\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-gm44fm/operationStatuses/08584276167533741052\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070251Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '44bd836b-527a-43f9-bbba-dd2564a97a3c', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070252Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0e93ac46-2e42-40bd-a4a6-ef9e55d0dd13', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0013c76e0>{ msg: "fai... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:158]: Unexpected error: <fmt.wrapError \| 0xc0013c7720>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-gm44fm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-gm44fm/operationStatuses/08584276167533741052 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T070251Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '44bd836b-527a-43f9-bbba-dd2564a97a3c', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\"\r\n }\r\n}" }, { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T070252Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0e93ac46-2e42-40bd-a4a6-ef9e55d0dd13', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-gm44fm\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-gm44fm/operationStatuses/08584276167533741052\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070251Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '44bd836b-527a-43f9-bbba-dd2564a97a3c', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070252Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0e93ac46-2e42-40bd-a4a6-ef9e55d0dd13', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0013c7700>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-node-limit-cluster-gm44fm\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-gm44fm/operationStatuses/08584276167533741052\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070251Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '44bd836b-527a-43f9-bbba-dd2564a97a3c', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\\\"\\r\\n }\\r\\n}\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070252Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0e93ac46-2e42-40bd-a4a6-ef9e55d0dd13', request correlation id 'ae924b17-2796-4f41-8cee-19775c28d9c7'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0013c76e0>{ msg: "fai... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 06:48:51 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:89]: Unexpected error: <fmt.wrapError \| 0xc001494f80>: failed to create customer-infra: failed waiting for deployment "customer-infra-cidr-connectivity-test-lp79ct" in resourcegroup="e2e-cidr-connectivity-z2nbjh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-z2nbjh/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-lp79ct/operationStatuses/08584276170570433662 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T065725Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0710dd8b-ba1f-43a2-bb18-1acc023a6668', request correlation id 'a5195d10-dc82-4d18-95a7-e17204226d83'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cidr-connectivity-test-lp79ct\" in resourcegroup=\"e2e-cidr-connectivity-z2nbjh\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-z2nbjh/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-lp79ct/operationStatuses/08584276170570433662\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T065725Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0710dd8b-ba1f-43a2-bb18-1acc023a6668', request correlation id 'a5195d10-dc82-4d18-95a7-e17204226d83'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001494f60>{ msg: "failed waiting for deployment \"customer-infra-cidr-connectivity-test-lp79ct\" in resourcegroup=\"e2e-cidr-connectivity-z2nbjh\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-z2nbjh/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-lp79ct/operationStatuses/08584276170570433662\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T065725Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0710dd8b-ba1f-43a2-bb18-1acc023a6668', request correlation id 'a5195d10-dc82-4d18-95a7-e17204226d83'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0013cc420>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["655"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Content-Type-Options": ["nosniff"], "Date": [ "Fri, 20 Mar 2026 06:57:31 GMT", ], "Pragma": ["no-cache"], "Content-Type": [ "application/json; charset=utf-8", ], "X-Ms-R... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:89]: Unexpected error: <fmt.wrapError \| 0xc001494f80>: failed to create customer-infra: failed waiting for deployment "customer-infra-cidr-connectivity-test-lp79ct" in resourcegroup="e2e-cidr-connectivity-z2nbjh" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-z2nbjh/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-lp79ct/operationStatuses/08584276170570433662 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T065725Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0710dd8b-ba1f-43a2-bb18-1acc023a6668', request correlation id 'a5195d10-dc82-4d18-95a7-e17204226d83'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cidr-connectivity-test-lp79ct\" in resourcegroup=\"e2e-cidr-connectivity-z2nbjh\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-z2nbjh/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-lp79ct/operationStatuses/08584276170570433662\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T065725Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0710dd8b-ba1f-43a2-bb18-1acc023a6668', request correlation id 'a5195d10-dc82-4d18-95a7-e17204226d83'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001494f60>{ msg: "failed waiting for deployment \"customer-infra-cidr-connectivity-test-lp79ct\" in resourcegroup=\"e2e-cidr-connectivity-z2nbjh\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-z2nbjh/providers/Microsoft.Resources/deployments/customer-infra-cidr-connectivity-test-lp79ct/operationStatuses/08584276170570433662\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T065725Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '0710dd8b-ba1f-43a2-bb18-1acc023a6668', request correlation id 'a5195d10-dc82-4d18-95a7-e17204226d83'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0013cc420>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["655"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Content-Type-Options": ["nosniff"], "Date": [ "Fri, 20 Mar 2026 06:57:31 GMT", ], "Pragma": ["no-cache"], "Content-Type": [ "application/json; charset=utf-8", ], "X-Ms-R... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 06:48:51 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error: <fmt.wrapError \| 0xc000f95e00>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-bsz975" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bsz975/operationStatuses/08584276169211157676 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T070005Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '972ea2fc-0260-4f89-a656-dc8b6e0e995f', request correlation id 'a59036d6-ee20-4527-867e-8ab3b02886fe'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-bsz975\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bsz975/operationStatuses/08584276169211157676\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070005Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '972ea2fc-0260-4f89-a656-dc8b6e0e995f', request correlation id 'a59036d6-ee20-4527-867e-8ab3b02886fe'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f95de0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-bsz975\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bsz975/operationStatuses/08584276169211157676\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070005Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '972ea2fc-0260-4f89-a656-dc8b6e0e995f', request correlation id 'a59036d6-ee20-4527-867e-8ab3b02886fe'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f95da0>{ msg: "failed waiting for deployment \"mi-np-autoscale-cluster-bsz975\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bsz975/operationStatuses/08584276169211157676\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error: <fmt.wrapError \| 0xc000f95e00>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-bsz975" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bsz975/operationStatuses/08584276169211157676 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T070005Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '972ea2fc-0260-4f89-a656-dc8b6e0e995f', request correlation id 'a59036d6-ee20-4527-867e-8ab3b02886fe'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-bsz975\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bsz975/operationStatuses/08584276169211157676\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070005Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '972ea2fc-0260-4f89-a656-dc8b6e0e995f', request correlation id 'a59036d6-ee20-4527-867e-8ab3b02886fe'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f95de0>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-np-autoscale-cluster-bsz975\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bsz975/operationStatuses/08584276169211157676\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070005Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '972ea2fc-0260-4f89-a656-dc8b6e0e995f', request correlation id 'a59036d6-ee20-4527-867e-8ab3b02886fe'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000f95da0>{ msg: "failed waiting for deployment \"mi-np-autoscale-cluster-bsz975\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bsz975/operationStatuses/08584276169211157676\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code InternalServerError 06:48:51 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:81]: Unexpected error: <fmt.wrapError \| 0xc00095b960>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-mmbdb4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mmbdb4/operationStatuses/08584276165787774143 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T070545Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '60d08050-88a9-4648-90df-f02523ca8c2b', request correlation id '383afc7c-1a46-4d7d-9386-066f9445e56f'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-mmbdb4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mmbdb4/operationStatuses/08584276165787774143\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070545Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '60d08050-88a9-4648-90df-f02523ca8c2b', request correlation id '383afc7c-1a46-4d7d-9386-066f9445e56f'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00095b940>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-mmbdb4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mmbdb4/operationStatuses/08584276165787774143\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070545Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '60d08050-88a9-4648-90df-f02523ca8c2b', request correlation id '383afc7c-1a46-4d7d-9386-066f9445e56f'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00095b8e0>{ msg: "failed waiting for deployment \"mi-negative-tests-cluster-mmbdb4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mmbdb4/operationStatuses/08584276165787774143\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploy... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:81]: Unexpected error: <fmt.wrapError \| 0xc00095b960>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-mmbdb4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mmbdb4/operationStatuses/08584276165787774143 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "InternalServerError", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"Encountered internal server error. Diagnostic information: timestamp '20260320T070545Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '60d08050-88a9-4648-90df-f02523ca8c2b', request correlation id '383afc7c-1a46-4d7d-9386-066f9445e56f'.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create managed identities: failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-mmbdb4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mmbdb4/operationStatuses/08584276165787774143\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070545Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '60d08050-88a9-4648-90df-f02523ca8c2b', request correlation id '383afc7c-1a46-4d7d-9386-066f9445e56f'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00095b940>{ msg: "failed to create managed identities: failed waiting for deployment \"mi-negative-tests-cluster-mmbdb4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mmbdb4/operationStatuses/08584276165787774143\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260320T070545Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '60d08050-88a9-4648-90df-f02523ca8c2b', request correlation id '383afc7c-1a46-4d7d-9386-066f9445e56f'.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00095b8e0>{ msg: "failed waiting for deployment \"mi-negative-tests-cluster-mmbdb4\" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-mmbdb4/operationStatuses/08584276165787774143\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploy... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
02:00:51	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a