CIHealth Run Log

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
19:15:22	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
15:55:39	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
11:44:43	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	43	e2e (43) ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-21-snpq9s" Unexpected error: <fmt.wrapError \| 0xc000c8c9a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-21-snpq9s-jlrh9l" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-21-snpq9s-jlrh9l/operationStatuses/08584187856226833932 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5vdsr5/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-uedd433jn6rrw\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-21-snpq9s" Unexpected error: <fmt.wrapError \| 0xc000c8c9a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-21-snpq9s-jlrh9l" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-21-snpq9s-jlrh9l/operationStatuses/08584187856226833932 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5vdsr5/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-uedd433jn6rrw\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:89]: failed to create customer resources for IDMS cluster Unexpected error: <fmt.wrapError \| 0xc000a50e20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-idms-e2e-hcp-cluster-vhb966" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-idms-e2e-hcp-cluster-vhb966/operationStatuses/08584187856875903212 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-d2rhb9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-lv7dvzspd2e5i\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:89]: failed to create customer resources for IDMS cluster Unexpected error: <fmt.wrapError \| 0xc000a50e20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-idms-e2e-hcp-cluster-vhb966" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-idms-e2e-hcp-cluster-vhb966/operationStatuses/08584187856875903212 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-d2rhb9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-lv7dvzspd2e5i\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000cfabc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-tfkh4v-kl9s9t" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-tfkh4v-kl9s9t/operationStatuses/08584187856742704984 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tfkh4v-r7jp9r/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-d6kl6v5o766mm\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000cfabc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-tfkh4v-kl9s9t" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-tfkh4v-kl9s9t/operationStatuses/08584187856742704984 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tfkh4v-r7jp9r/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-d6kl6v5o766mm\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:579]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000e08ca0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-downgrade-w5r6p4-thj4kq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-downgrade-w5r6p4-thj4kq/operationStatuses/08584187855779410769 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-w5r6p4-cqjc9p/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-li3mbgo574w7y\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:579]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000e08ca0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-downgrade-w5r6p4-thj4kq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-downgrade-w5r6p4-thj4kq/operationStatuses/08584187855779410769 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-w5r6p4-cqjc9p/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-li3mbgo574w7y\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:207]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc0010740c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-oidc-wi-cluster-g8tpd4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-oidc-wi-cluster-g8tpd4/operationStatuses/08584187855657382633 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gt4j9v/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-s6kwp7xg43dbi\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:207]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc0010740c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-oidc-wi-cluster-g8tpd4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-oidc-wi-cluster-g8tpd4/operationStatuses/08584187855657382633 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gt4j9v/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-s6kwp7xg43dbi\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: failed to create customer resources for external auth cluster Unexpected error: <fmt.wrapError \| 0xc0009cac60>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-cluster-cvvms6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-cvvms6/operationStatuses/08584187856875515968 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-9g6rx9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ltdphft7fgrwg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: failed to create customer resources for external auth cluster Unexpected error: <fmt.wrapError \| 0xc0009cac60>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-cluster-cvvms6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-cvvms6/operationStatuses/08584187856875515968 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-9g6rx9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ltdphft7fgrwg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:79]: failed to create customer resources for cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc0006ee960>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cx-rg-hcp-cluster-h9tz2c" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cx-rg-hcp-cluster-h9tz2c/operationStatuses/08584187858039535675 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-wwvf8k/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fmxs36wu5sow2\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:79]: failed to create customer resources for cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc0006ee960>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cx-rg-hcp-cluster-h9tz2c" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cx-rg-hcp-cluster-h9tz2c/operationStatuses/08584187858039535675 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-wwvf8k/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fmxs36wu5sow2\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-19-j86mr5" Unexpected error: <fmt.wrapError \| 0xc000dab380>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-j86mr5-h8dzxs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-j86mr5-h8dzxs/operationStatuses/08584187855615521778 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-8bcxrs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-tzhrgbwizlkbk\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-19-j86mr5" Unexpected error: <fmt.wrapError \| 0xc000dab380>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-j86mr5-h8dzxs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-j86mr5-h8dzxs/operationStatuses/08584187855615521778 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-8bcxrs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-tzhrgbwizlkbk\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:97]: failed to create customer resources for basic-cluster Unexpected error: <fmt.wrapError \| 0xc00060d580>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-cluster-hdl2hm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-hdl2hm/operationStatuses/08584187857936124773 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-7pfrcv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j3gkygvtvkvjs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:97]: failed to create customer resources for basic-cluster Unexpected error: <fmt.wrapError \| 0xc00060d580>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-cluster-hdl2hm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-hdl2hm/operationStatuses/08584187857936124773 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-7pfrcv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j3gkygvtvkvjs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:117]: failed to deploy managed identities for back-level version 4.19 Unexpected error: <fmt.wrapError \| 0xc0013808e0>: failed waiting for deployment "mi-cluster-ver-4-19-zk7b56" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-ver-4-19-zk7b56/operationStatuses/08584187858039921366 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-8grkk8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:117]: failed to deploy managed identities for back-level version 4.19 Unexpected error: <fmt.wrapError \| 0xc0013808e0>: failed waiting for deployment "mi-cluster-ver-4-19-zk7b56" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-ver-4-19-zk7b56/operationStatuses/08584187858039921366 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-8grkk8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:102]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000620540>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ephemeral-disk-2k6k98" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ephemeral-disk-2k6k98/operationStatuses/08584187856873136686 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-zrh7v9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6xts4ycasve7q\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:102]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000620540>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ephemeral-disk-2k6k98" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ephemeral-disk-2k6k98/operationStatuses/08584187856873136686 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-zrh7v9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6xts4ycasve7q\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:73]: failed to create customer resources for cilium CNI cluster Unexpected error: <fmt.wrapError \| 0xc0010aecc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cilium-cl-fxw9ws" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cilium-cl-fxw9ws/operationStatuses/08584187857935123531 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zgvs8z/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-dve5f7wncegny\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:73]: failed to create customer resources for cilium CNI cluster Unexpected error: <fmt.wrapError \| 0xc0010aecc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cilium-cl-fxw9ws" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cilium-cl-fxw9ws/operationStatuses/08584187857935123531 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zgvs8z/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-dve5f7wncegny\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:77]: failed to create customer resources for ARM64 nodepool cluster Unexpected error: <fmt.wrapError \| 0xc000f093a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-arm64-vm-hcp-cluster-snvg8v" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-arm64-vm-hcp-cluster-snvg8v/operationStatuses/08584187858035517963 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-5gslgw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6htyoibevj7ye\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:77]: failed to create customer resources for ARM64 nodepool cluster Unexpected error: <fmt.wrapError \| 0xc000f093a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-arm64-vm-hcp-cluster-snvg8v" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-arm64-vm-hcp-cluster-snvg8v/operationStatuses/08584187858035517963 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-5gslgw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6htyoibevj7ye\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-20-8wbp2c" Unexpected error: <fmt.wrapError \| 0xc000e0f220>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-20-8wbp2c-m72l5v" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-20-8wbp2c-m72l5v/operationStatuses/08584187855622714364 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-vcmx9c/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-yho4o5rtioo24\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-20-8wbp2c" Unexpected error: <fmt.wrapError \| 0xc000e0f220>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-20-8wbp2c-m72l5v" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-20-8wbp2c-m72l5v/operationStatuses/08584187855622714364 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-vcmx9c/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-yho4o5rtioo24\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:103]: failed to create customer resources for pull secret cluster Unexpected error: <fmt.wrapError \| 0xc00092d060>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-pullsecret-hcp-cluster-h6xfln" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-pullsecret-hcp-cluster-h6xfln/operationStatuses/08584187857890138763 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-2lsrl6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-p2h32vyw2l2ma\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:103]: failed to create customer resources for pull secret cluster Unexpected error: <fmt.wrapError \| 0xc00092d060>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-pullsecret-hcp-cluster-h6xfln" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-pullsecret-hcp-cluster-h6xfln/operationStatuses/08584187857890138763 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-2lsrl6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-p2h32vyw2l2ma\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:90]: failed to create customer resources for authorized CIDRs cluster Unexpected error: <fmt.wrapError \| 0xc00080bbc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-9wrxj4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-9wrxj4/operationStatuses/08584187858033619638 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-75rvdf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:90]: failed to create customer resources for authorized CIDRs cluster Unexpected error: <fmt.wrapError \| 0xc00080bbc0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-9wrxj4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-9wrxj4/operationStatuses/08584187858033619638 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-75rvdf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-22-vq6vk4" Unexpected error: <fmt.wrapError \| 0xc000d10940>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-22-vq6vk4-5tjqb7" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-22-vq6vk4-5tjqb7/operationStatuses/08584187856828514260 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-q8wmpl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-pfswajy6jao2s\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-22-vq6vk4" Unexpected error: <fmt.wrapError \| 0xc000d10940>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-22-vq6vk4-5tjqb7" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-22-vq6vk4-5tjqb7/operationStatuses/08584187856828514260 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-q8wmpl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-pfswajy6jao2s\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: failed to create customer resources for first cluster "basic-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000d07940>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-znkv24" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-znkv24/operationStatuses/08584187858039738005 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-5cstzv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: failed to create customer resources for first cluster "basic-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000d07940>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-znkv24" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-znkv24/operationStatuses/08584187858039738005 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-5cstzv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:83]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000d79000>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-h4sd57" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-h4sd57/operationStatuses/08584187855739788076 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-5fjq8b/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:83]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000d79000>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-h4sd57" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-h4sd57/operationStatuses/08584187855739788076 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-5fjq8b/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:77]: failed to create customer resources for external auth list test Unexpected error: <fmt.wrapError \| 0xc000b84b20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-listhq4n6c-wzns28" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listhq4n6c-wzns28/operationStatuses/08584187856776557038 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9ltlvd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-my6owol6p5vmm\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:77]: failed to create customer resources for external auth list test Unexpected error: <fmt.wrapError \| 0xc000b84b20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-listhq4n6c-wzns28" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listhq4n6c-wzns28/operationStatuses/08584187856776557038 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9ltlvd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-my6owol6p5vmm\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:449]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000ccd460>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-skip-minor-72szdl-fpzdhp" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-skip-minor-72szdl-fpzdhp/operationStatuses/08584187856560051514 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-72szdl-nh8g5v/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-jmzrlgk5vtvcc\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:449]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000ccd460>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-skip-minor-72szdl-fpzdhp" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-skip-minor-72szdl-fpzdhp/operationStatuses/08584187856560051514 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-72szdl-nh8g5v/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-jmzrlgk5vtvcc\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-21-dz2ll2" Unexpected error: <fmt.wrapError \| 0xc000a5bea0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-dz2ll2-wb6r7g" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-dz2ll2-wb6r7g/operationStatuses/08584187855570807578 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4bt6b9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xgsfx5evhbp6k\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-21-dz2ll2" Unexpected error: <fmt.wrapError \| 0xc000a5bea0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-dz2ll2-wb6r7g" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-dz2ll2-wb6r7g/operationStatuses/08584187855570807578 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4bt6b9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xgsfx5evhbp6k\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:74]: failed to create customer resources with private key vault Unexpected error: <fmt.wrapError \| 0xc000e0eac0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-private-kv-cluster-m8cdqs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-private-kv-cluster-m8cdqs/operationStatuses/08584187857019650480 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4klntz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-had2isw6vkgre\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:74]: failed to create customer resources with private key vault Unexpected error: <fmt.wrapError \| 0xc000e0eac0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-private-kv-cluster-m8cdqs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-private-kv-cluster-m8cdqs/operationStatuses/08584187857019650480 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4klntz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-had2isw6vkgre\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:149]: failed to create customer resources for patch-tags cluster Unexpected error: <fmt.wrapError \| 0xc001508da0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-rxqsg5" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-rxqsg5/operationStatuses/08584187858042570035 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-s7b5tf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:149]: failed to create customer resources for patch-tags cluster Unexpected error: <fmt.wrapError \| 0xc001508da0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-rxqsg5" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-rxqsg5/operationStatuses/08584187858042570035 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-s7b5tf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:81]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000e9d140>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-xpnlw6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-xpnlw6/operationStatuses/08584187856711496599 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-8tcqz7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:81]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000e9d140>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-xpnlw6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-xpnlw6/operationStatuses/08584187856711496599 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-8tcqz7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-22-tjhzrs" Unexpected error: <fmt.wrapError \| 0xc000d89ea0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-22-tjhzrs-sb4966" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-22-tjhzrs-sb4966/operationStatuses/08584187856891316107 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-jx9frs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wku3kkbhewpoe\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-22-tjhzrs" Unexpected error: <fmt.wrapError \| 0xc000d89ea0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-22-tjhzrs-sb4966" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-22-tjhzrs-sb4966/operationStatuses/08584187856891316107 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-jx9frs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wku3kkbhewpoe\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc0008c7f40>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-gz5klc-gl25fs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-gz5klc-gl25fs/operationStatuses/08584187856741962093 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-gz5klc-d5g8nh/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-hruihh5h6o7ho\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc0008c7f40>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-gz5klc-gl25fs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-gz5klc-gl25fs/operationStatuses/08584187856741962093 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-gz5klc-d5g8nh/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-hruihh5h6o7ho\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:73]: failed to create customer resources for patch-name cluster Unexpected error: <fmt.wrapError \| 0xc000b0e360>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-name-cluster-74x4gh" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-name-cluster-74x4gh/operationStatuses/08584187858034157061 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-85zpmt/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rr7tdjqoygobu\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:73]: failed to create customer resources for patch-name cluster Unexpected error: <fmt.wrapError \| 0xc000b0e360>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-name-cluster-74x4gh" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-name-cluster-74x4gh/operationStatuses/08584187858034157061 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-85zpmt/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rr7tdjqoygobu\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:88]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc0008af780>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-bxqzwd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bxqzwd/operationStatuses/08584187856854920273 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sv6z6b/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:88]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc0008af780>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-bxqzwd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bxqzwd/operationStatuses/08584187856854920273 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sv6z6b/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:726]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000b03a40>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-dg-minor-rmcnxc-hmljs9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-dg-minor-rmcnxc-hmljs9/operationStatuses/08584187855656406602 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-rmcnxc-v65txx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ubvi6leodlqa\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:726]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000b03a40>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-dg-minor-rmcnxc-hmljs9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-dg-minor-rmcnxc-hmljs9/operationStatuses/08584187855656406602 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-rmcnxc-v65txx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ubvi6leodlqa\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:88]: failed to create customer resources for autoscaling cluster Unexpected error: <fmt.wrapError \| 0xc000e118e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-autoscaling-hcp-cluster-htvsqw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-htvsqw/operationStatuses/08584187858022867554 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws9p9j/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:88]: failed to create customer resources for autoscaling cluster Unexpected error: <fmt.wrapError \| 0xc000e118e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-autoscaling-hcp-cluster-htvsqw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-htvsqw/operationStatuses/08584187858022867554 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws9p9j/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc0009e7c20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-noedge-px4p2g-4l884k" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-noedge-px4p2g-4l884k/operationStatuses/08584187856731725299 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-px4p2g-q6krz9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-mnlsiq3q4kxek\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc0009e7c20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-noedge-px4p2g-4l884k" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-noedge-px4p2g-4l884k/operationStatuses/08584187856731725299 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-px4p2g-q6krz9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-mnlsiq3q4kxek\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc00100a6e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-4ltht9-7j6qsx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-4ltht9-7j6qsx/operationStatuses/08584187856773401940 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-4ltht9-5jkm6t/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xb3lnijigxmoa\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc00100a6e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-4ltht9-7j6qsx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-4ltht9-7j6qsx/operationStatuses/08584187856773401940 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-4ltht9-5jkm6t/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xb3lnijigxmoa\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-5-0-9ldzcl" Unexpected error: <fmt.wrapError \| 0xc000e973a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-5-0-9ldzcl-xl49gs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-5-0-9ldzcl-xl49gs/operationStatuses/08584187856834194188 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-sfjl8m/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-io4laudmh4wje\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-5-0-9ldzcl" Unexpected error: <fmt.wrapError \| 0xc000e973a0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-5-0-9ldzcl-xl49gs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-5-0-9ldzcl-xl49gs/operationStatuses/08584187856834194188 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-sfjl8m/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-io4laudmh4wje\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:70]: failed to create customer resources for cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc000f104e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-hcp-cluster-np-128-lp87gt" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-hcp-cluster-np-128-lp87gt/operationStatuses/08584187858042127885 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-mzr59f/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ljyjjx2n7gxo\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:70]: failed to create customer resources for cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc000f104e0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-hcp-cluster-np-128-lp87gt" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-hcp-cluster-np-128-lp87gt/operationStatuses/08584187858042127885 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-mzr59f/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ljyjjx2n7gxo\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test HCP Nodepools GPU instances creates and deletes a GPU nodepool in a single cluster · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:72]: failed to create customer resources for GPU nodepool cluster Unexpected error: <fmt.wrapError \| 0xc001a0c2c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-gpu-np-cp7dkx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-gpu-np-cp7dkx/operationStatuses/08584187856780553726 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-jqpcdg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ruz4uxlifk3nm\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:72]: failed to create customer resources for GPU nodepool cluster Unexpected error: <fmt.wrapError \| 0xc001a0c2c0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-gpu-np-cp7dkx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-gpu-np-cp7dkx/operationStatuses/08584187856780553726 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-jqpcdg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ruz4uxlifk3nm\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-20-vh49qb" Unexpected error: <fmt.wrapError \| 0xc001281560>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-vh49qb-9mp8fg" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-vh49qb-9mp8fg/operationStatuses/08584187855617573436 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-5rdrhb/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ks6edxnijirju\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-20-vh49qb" Unexpected error: <fmt.wrapError \| 0xc001281560>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-vh49qb-9mp8fg" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-vh49qb-9mp8fg/operationStatuses/08584187855617573436 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-5rdrhb/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ks6edxnijirju\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:78]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000514c20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-j5nhqc" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-j5nhqc/operationStatuses/08584187856862566654 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-xm4sj4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:78]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0xc000514c20>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-j5nhqc" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-j5nhqc/operationStatuses/08584187856862566654 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-xm4sj4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:276]: failed to create cluster customer resources for node-limit test Unexpected error: <fmt.wrapError \| 0xc000b1a640>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-hxvbhg" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-hxvbhg/operationStatuses/08584187856876786135 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-7kzpgt/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:276]: failed to create cluster customer resources for node-limit test Unexpected error: <fmt.wrapError \| 0xc000b1a640>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-hxvbhg" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-hxvbhg/operationStatuses/08584187856876786135 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-7kzpgt/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:85]: failed to create customer resources for TLS endpoint cluster Unexpected error: <fmt.wrapError \| 0xc000f84de0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-tls-endpoint-hcp-cluster-nkdnjx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-nkdnjx/operationStatuses/08584187858040367930 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-vbsggw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oo7pyvaeowk44\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:85]: failed to create customer resources for TLS endpoint cluster Unexpected error: <fmt.wrapError \| 0xc000f84de0>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-tls-endpoint-hcp-cluster-nkdnjx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-nkdnjx/operationStatuses/08584187858040367930 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-vbsggw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oo7pyvaeowk44\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: FailedIdentityOperation; provider Microsoft.ManagedIdentity 11:44:43 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:132]: failed to start cluster creation Unexpected error: <exported.ResponseError \| 0xc000a623c0>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-dcqfgd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: FailedIdentityOperation -------------------------------------------------------------------------------- { "error": { "code": "FailedIdentityOperation", "message": "Identity operation for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-dcqfgd/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/delayed-rbac-cluster' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/aro-hcp-msi-container-int-shard0-00-13/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager' was not found.\"}}'.'." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:132]: failed to start cluster creation Unexpected error: <exported.ResponseError \| 0xc000a623c0>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-dcqfgd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: FailedIdentityOperation -------------------------------------------------------------------------------- { "error": { "code": "FailedIdentityOperation", "message": "Identity operation for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-dcqfgd/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/delayed-rbac-cluster' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/aro-hcp-msi-container-int-shard0-00-13/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager' was not found.\"}}'.'." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:81]: failed to create customer resources with private key vault Unexpected error: <fmt.wrapError \| 0xc000c04500>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cilium-cluster-bknd4f" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cilium-cluster-bknd4f/operationStatuses/08584187857020344075 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4gslc2/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-4vnk4izirdfzu\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:81]: failed to create customer resources with private key vault Unexpected error: <fmt.wrapError \| 0xc000c04500>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cilium-cluster-bknd4f" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cilium-cluster-bknd4f/operationStatuses/08584187857020344075 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4gslc2/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-4vnk4izirdfzu\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found. 11:44:43 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:87]: failed to create customer resources for admin credential lifecycle cluster Unexpected error: <fmt.wrapError \| 0xc000f9ec40>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-admin-cred-lifecycle-pnnk7t-pb8dz9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-pnnk7t-pb8dz9/operationStatuses/08584187858025314704 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-9hbvc5/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-es7labf5uzmr2\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:87]: failed to create customer resources for admin credential lifecycle cluster Unexpected error: <fmt.wrapError \| 0xc000f9ec40>: failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-admin-cred-lifecycle-pnnk7t-pb8dz9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-pnnk7t-pb8dz9/operationStatuses/08584187858025314704 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Conflict", "message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-9hbvc5/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-es7labf5uzmr2\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n },\r\n {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}" }, { "code": "NotFound", "message": "{\r\n \"error\": {\r\n \"code\": \"ResourceGroupNotFound\",\r\n \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- ... occurred
07:13:08	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
04:22:15	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-21-snpq9s"
Unexpected error:
    <*fmt.wrapError | 0xc000c8c9a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-21-snpq9s-jlrh9l" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-21-snpq9s-jlrh9l/operationStatuses/08584187856226833932
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5vdsr5/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-uedd433jn6rrw\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-21-snpq9s"
Unexpected error:
    <*fmt.wrapError | 0xc000c8c9a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-21-snpq9s-jlrh9l" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-21-snpq9s-jlrh9l/operationStatuses/08584187856226833932
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-5vdsr5/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-uedd433jn6rrw\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:89]: failed to create customer resources for IDMS cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a50e20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-idms-e2e-hcp-cluster-vhb966" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-idms-e2e-hcp-cluster-vhb966/operationStatuses/08584187856875903212
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-d2rhb9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-lv7dvzspd2e5i\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:89]: failed to create customer resources for IDMS cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a50e20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-idms-e2e-hcp-cluster-vhb966" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-idms-e2e-hcp-cluster-vhb966/operationStatuses/08584187856875903212
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-d2rhb9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-lv7dvzspd2e5i\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000cfabc0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-tfkh4v-kl9s9t" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-tfkh4v-kl9s9t/operationStatuses/08584187856742704984
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tfkh4v-r7jp9r/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-d6kl6v5o766mm\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000cfabc0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-tfkh4v-kl9s9t" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-tfkh4v-kl9s9t/operationStatuses/08584187856742704984
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tfkh4v-r7jp9r/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-d6kl6v5o766mm\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:579]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000e08ca0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-downgrade-w5r6p4-thj4kq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-downgrade-w5r6p4-thj4kq/operationStatuses/08584187855779410769
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-w5r6p4-cqjc9p/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-li3mbgo574w7y\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:579]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000e08ca0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-downgrade-w5r6p4-thj4kq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-downgrade-w5r6p4-thj4kq/operationStatuses/08584187855779410769
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-w5r6p4-cqjc9p/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-li3mbgo574w7y\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:207]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0010740c0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-oidc-wi-cluster-g8tpd4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-oidc-wi-cluster-g8tpd4/operationStatuses/08584187855657382633
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gt4j9v/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-s6kwp7xg43dbi\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:207]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0010740c0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-oidc-wi-cluster-g8tpd4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-oidc-wi-cluster-g8tpd4/operationStatuses/08584187855657382633
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gt4j9v/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-s6kwp7xg43dbi\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: failed to create customer resources for external auth cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009cac60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-cluster-cvvms6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-cvvms6/operationStatuses/08584187856875515968
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-9g6rx9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ltdphft7fgrwg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: failed to create customer resources for external auth cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009cac60>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-cluster-cvvms6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-cvvms6/operationStatuses/08584187856875515968
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-9g6rx9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ltdphft7fgrwg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:79]: failed to create customer resources for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0006ee960>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cx-rg-hcp-cluster-h9tz2c" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cx-rg-hcp-cluster-h9tz2c/operationStatuses/08584187858039535675
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-wwvf8k/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fmxs36wu5sow2\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:79]: failed to create customer resources for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0006ee960>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cx-rg-hcp-cluster-h9tz2c" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cx-rg-hcp-cluster-h9tz2c/operationStatuses/08584187858039535675
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-wwvf8k/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fmxs36wu5sow2\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.19 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-19-j86mr5"
Unexpected error:
    <*fmt.wrapError | 0xc000dab380>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-j86mr5-h8dzxs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-j86mr5-h8dzxs/operationStatuses/08584187855615521778
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-8bcxrs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-tzhrgbwizlkbk\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-19-j86mr5"
Unexpected error:
    <*fmt.wrapError | 0xc000dab380>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-19-j86mr5-h8dzxs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-19-j86mr5-h8dzxs/operationStatuses/08584187855615521778
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-8bcxrs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-tzhrgbwizlkbk\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:97]: failed to create customer resources for basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00060d580>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-cluster-hdl2hm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-hdl2hm/operationStatuses/08584187857936124773
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-7pfrcv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j3gkygvtvkvjs\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:97]: failed to create customer resources for basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00060d580>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-cluster-hdl2hm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-cluster-hdl2hm/operationStatuses/08584187857936124773
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-7pfrcv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j3gkygvtvkvjs\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:117]: failed to deploy managed identities for back-level version 4.19
Unexpected error:
    <*fmt.wrapError | 0xc0013808e0>: 
    failed waiting for deployment "mi-cluster-ver-4-19-zk7b56" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-ver-4-19-zk7b56/operationStatuses/08584187858039921366
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-8grkk8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:117]: failed to deploy managed identities for back-level version 4.19
Unexpected error:
    <*fmt.wrapError | 0xc0013808e0>: 
    failed waiting for deployment "mi-cluster-ver-4-19-zk7b56" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-ver-4-19-zk7b56/operationStatuses/08584187858039921366
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-8grkk8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:102]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000620540>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ephemeral-disk-2k6k98" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ephemeral-disk-2k6k98/operationStatuses/08584187856873136686
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-zrh7v9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6xts4ycasve7q\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:102]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000620540>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ephemeral-disk-2k6k98" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ephemeral-disk-2k6k98/operationStatuses/08584187856873136686
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-zrh7v9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6xts4ycasve7q\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:73]: failed to create customer resources for cilium CNI cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010aecc0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cilium-cl-fxw9ws" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cilium-cl-fxw9ws/operationStatuses/08584187857935123531
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zgvs8z/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-dve5f7wncegny\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:73]: failed to create customer resources for cilium CNI cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010aecc0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cilium-cl-fxw9ws" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cilium-cl-fxw9ws/operationStatuses/08584187857935123531
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-zgvs8z/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-dve5f7wncegny\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:77]: failed to create customer resources for ARM64 nodepool cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f093a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-arm64-vm-hcp-cluster-snvg8v" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-arm64-vm-hcp-cluster-snvg8v/operationStatuses/08584187858035517963
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-5gslgw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6htyoibevj7ye\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:77]: failed to create customer resources for ARM64 nodepool cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f093a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-arm64-vm-hcp-cluster-snvg8v" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-arm64-vm-hcp-cluster-snvg8v/operationStatuses/08584187858035517963
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-5gslgw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6htyoibevj7ye\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-20-8wbp2c"
Unexpected error:
    <*fmt.wrapError | 0xc000e0f220>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-20-8wbp2c-m72l5v" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-20-8wbp2c-m72l5v/operationStatuses/08584187855622714364
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-vcmx9c/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-yho4o5rtioo24\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-20-8wbp2c"
Unexpected error:
    <*fmt.wrapError | 0xc000e0f220>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-20-8wbp2c-m72l5v" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-20-8wbp2c-m72l5v/operationStatuses/08584187855622714364
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-vcmx9c/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-yho4o5rtioo24\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:103]: failed to create customer resources for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc00092d060>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-pullsecret-hcp-cluster-h6xfln" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-pullsecret-hcp-cluster-h6xfln/operationStatuses/08584187857890138763
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-2lsrl6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-p2h32vyw2l2ma\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:103]: failed to create customer resources for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc00092d060>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-pullsecret-hcp-cluster-h6xfln" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-pullsecret-hcp-cluster-h6xfln/operationStatuses/08584187857890138763
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-2lsrl6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-p2h32vyw2l2ma\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:90]: failed to create customer resources for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc00080bbc0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-9wrxj4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-9wrxj4/operationStatuses/08584187858033619638
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-75rvdf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:90]: failed to create customer resources for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc00080bbc0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cidr-connectivity-test-9wrxj4" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cidr-connectivity-test-9wrxj4/operationStatuses/08584187858033619638
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-75rvdf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wihoigd7cpguy\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-22-vq6vk4"
Unexpected error:
    <*fmt.wrapError | 0xc000d10940>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-22-vq6vk4-5tjqb7" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-22-vq6vk4-5tjqb7/operationStatuses/08584187856828514260
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-q8wmpl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-pfswajy6jao2s\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-4-22-vq6vk4"
Unexpected error:
    <*fmt.wrapError | 0xc000d10940>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-4-22-vq6vk4-5tjqb7" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-4-22-vq6vk4-5tjqb7/operationStatuses/08584187856828514260
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-q8wmpl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-pfswajy6jao2s\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: failed to create customer resources for first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000d07940>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-znkv24" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-znkv24/operationStatuses/08584187858039738005
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-5cstzv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: failed to create customer resources for first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000d07940>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-basic-hcp-cluster-znkv24" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-basic-hcp-cluster-znkv24/operationStatuses/08584187858039738005
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-5cstzv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-eqbgltuyzhpg4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:83]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000d79000>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-h4sd57" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-h4sd57/operationStatuses/08584187855739788076
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-5fjq8b/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:83]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000d79000>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-h4sd57" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-h4sd57/operationStatuses/08584187855739788076
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-5fjq8b/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:77]: failed to create customer resources for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000b84b20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-listhq4n6c-wzns28" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listhq4n6c-wzns28/operationStatuses/08584187856776557038
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9ltlvd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-my6owol6p5vmm\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:77]: failed to create customer resources for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000b84b20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-listhq4n6c-wzns28" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listhq4n6c-wzns28/operationStatuses/08584187856776557038
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9ltlvd/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-my6owol6p5vmm\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-02' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:449]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000ccd460>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-skip-minor-72szdl-fpzdhp" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-skip-minor-72szdl-fpzdhp/operationStatuses/08584187856560051514
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-72szdl-nh8g5v/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-jmzrlgk5vtvcc\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:449]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000ccd460>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-skip-minor-72szdl-fpzdhp" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-skip-minor-72szdl-fpzdhp/operationStatuses/08584187856560051514
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-72szdl-nh8g5v/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-jmzrlgk5vtvcc\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-15' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-21-dz2ll2"
Unexpected error:
    <*fmt.wrapError | 0xc000a5bea0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-dz2ll2-wb6r7g" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-dz2ll2-wb6r7g/operationStatuses/08584187855570807578
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4bt6b9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xgsfx5evhbp6k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-21-dz2ll2"
Unexpected error:
    <*fmt.wrapError | 0xc000a5bea0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-dz2ll2-wb6r7g" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-dz2ll2-wb6r7g/operationStatuses/08584187855570807578
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4bt6b9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xgsfx5evhbp6k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:74]: failed to create customer resources with private key vault
Unexpected error:
    <*fmt.wrapError | 0xc000e0eac0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-private-kv-cluster-m8cdqs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-private-kv-cluster-m8cdqs/operationStatuses/08584187857019650480
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4klntz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-had2isw6vkgre\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:74]: failed to create customer resources with private key vault
Unexpected error:
    <*fmt.wrapError | 0xc000e0eac0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-private-kv-cluster-m8cdqs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-private-kv-cluster-m8cdqs/operationStatuses/08584187857019650480
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4klntz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-had2isw6vkgre\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-09' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:149]: failed to create customer resources for patch-tags cluster
Unexpected error:
    <*fmt.wrapError | 0xc001508da0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-rxqsg5" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-rxqsg5/operationStatuses/08584187858042570035
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-s7b5tf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:149]: failed to create customer resources for patch-tags cluster
Unexpected error:
    <*fmt.wrapError | 0xc001508da0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-tags-cluster-rxqsg5" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-tags-cluster-rxqsg5/operationStatuses/08584187858042570035
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-s7b5tf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-fklo75htpxzxc\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-04' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:81]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000e9d140>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-xpnlw6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-xpnlw6/operationStatuses/08584187856711496599
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-8tcqz7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:81]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000e9d140>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-xpnlw6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-xpnlw6/operationStatuses/08584187856711496599
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-8tcqz7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-22-tjhzrs"
Unexpected error:
    <*fmt.wrapError | 0xc000d89ea0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-22-tjhzrs-sb4966" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-22-tjhzrs-sb4966/operationStatuses/08584187856891316107
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-jx9frs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wku3kkbhewpoe\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-22-tjhzrs"
Unexpected error:
    <*fmt.wrapError | 0xc000d89ea0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-22-tjhzrs-sb4966" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-22-tjhzrs-sb4966/operationStatuses/08584187856891316107
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-jx9frs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-wku3kkbhewpoe\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-13' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0008c7f40>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-gz5klc-gl25fs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-gz5klc-gl25fs/operationStatuses/08584187856741962093
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-gz5klc-d5g8nh/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-hruihh5h6o7ho\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0008c7f40>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-gz5klc-gl25fs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-gz5klc-gl25fs/operationStatuses/08584187856741962093
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-gz5klc-d5g8nh/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-hruihh5h6o7ho\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-11' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:73]: failed to create customer resources for patch-name cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b0e360>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-name-cluster-74x4gh" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-name-cluster-74x4gh/operationStatuses/08584187858034157061
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-85zpmt/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rr7tdjqoygobu\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:73]: failed to create customer resources for patch-name cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b0e360>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-patch-name-cluster-74x4gh" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-patch-name-cluster-74x4gh/operationStatuses/08584187858034157061
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-85zpmt/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rr7tdjqoygobu\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-16' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:88]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0008af780>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-bxqzwd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bxqzwd/operationStatuses/08584187856854920273
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sv6z6b/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:88]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0008af780>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-bxqzwd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-bxqzwd/operationStatuses/08584187856854920273
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sv6z6b/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-05' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:726]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000b03a40>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-dg-minor-rmcnxc-hmljs9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-dg-minor-rmcnxc-hmljs9/operationStatuses/08584187855656406602
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-rmcnxc-v65txx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ubvi6leodlqa\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:726]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000b03a40>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-dg-minor-rmcnxc-hmljs9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-dg-minor-rmcnxc-hmljs9/operationStatuses/08584187855656406602
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-rmcnxc-v65txx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ubvi6leodlqa\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-00' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:88]: failed to create customer resources for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e118e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-autoscaling-hcp-cluster-htvsqw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-htvsqw/operationStatuses/08584187858022867554
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws9p9j/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:88]: failed to create customer resources for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e118e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-autoscaling-hcp-cluster-htvsqw" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-htvsqw/operationStatuses/08584187858022867554
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ws9p9j/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-19' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0009e7c20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-noedge-px4p2g-4l884k" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-noedge-px4p2g-4l884k/operationStatuses/08584187856731725299
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-px4p2g-q6krz9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-mnlsiq3q4kxek\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:302]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0009e7c20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-noedge-px4p2g-4l884k" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-noedge-px4p2g-4l884k/operationStatuses/08584187856731725299
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-px4p2g-q6krz9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-mnlsiq3q4kxek\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-17' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc00100a6e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-4ltht9-7j6qsx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-4ltht9-7j6qsx/operationStatuses/08584187856773401940
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-4ltht9-5jkm6t/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xb3lnijigxmoa\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:122]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc00100a6e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-4ltht9-7j6qsx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-4ltht9-7j6qsx/operationStatuses/08584187856773401940
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-4ltht9-5jkm6t/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xb3lnijigxmoa\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-07' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-5-0-9ldzcl"
Unexpected error:
    <*fmt.wrapError | 0xc000e973a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-5-0-9ldzcl-xl49gs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-5-0-9ldzcl-xl49gs/operationStatuses/08584187856834194188
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-sfjl8m/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-io4laudmh4wje\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:95]: failed to create customer resources for cluster "cluster-candidate-5-0-9ldzcl"
Unexpected error:
    <*fmt.wrapError | 0xc000e973a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-candidate-5-0-9ldzcl-xl49gs" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-candidate-5-0-9ldzcl-xl49gs/operationStatuses/08584187856834194188
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-sfjl8m/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-io4laudmh4wje\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:70]: failed to create customer resources for cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000f104e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-hcp-cluster-np-128-lp87gt" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-hcp-cluster-np-128-lp87gt/operationStatuses/08584187858042127885
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-mzr59f/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ljyjjx2n7gxo\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:70]: failed to create customer resources for cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000f104e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-hcp-cluster-np-128-lp87gt" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-hcp-cluster-np-128-lp87gt/operationStatuses/08584187858042127885
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-mzr59f/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3ljyjjx2n7gxo\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-18' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test HCP Nodepools GPU instances creates and deletes a GPU nodepool in a single cluster · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:72]: failed to create customer resources for GPU nodepool cluster
Unexpected error:
    <*fmt.wrapError | 0xc001a0c2c0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-gpu-np-cp7dkx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-gpu-np-cp7dkx/operationStatuses/08584187856780553726
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-jqpcdg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ruz4uxlifk3nm\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:72]: failed to create customer resources for GPU nodepool cluster
Unexpected error:
    <*fmt.wrapError | 0xc001a0c2c0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-gpu-np-cp7dkx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-gpu-np-cp7dkx/operationStatuses/08584187856780553726
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-gpu-nodepool-jqpcdg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ruz4uxlifk3nm\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-06' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-20-vh49qb"
Unexpected error:
    <*fmt.wrapError | 0xc001281560>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-vh49qb-9mp8fg" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-vh49qb-9mp8fg/operationStatuses/08584187855617573436
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-5rdrhb/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ks6edxnijirju\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-20-vh49qb"
Unexpected error:
    <*fmt.wrapError | 0xc001281560>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-vh49qb-9mp8fg" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-vh49qb-9mp8fg/operationStatuses/08584187855617573436
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-5rdrhb/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ks6edxnijirju\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-10' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:78]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000514c20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-j5nhqc" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-j5nhqc/operationStatuses/08584187856862566654
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-xm4sj4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:78]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000514c20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-j5nhqc" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-j5nhqc/operationStatuses/08584187856862566654
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-xm4sj4/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-01' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:276]: failed to create cluster customer resources for node-limit test
Unexpected error:
    <*fmt.wrapError | 0xc000b1a640>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-hxvbhg" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-hxvbhg/operationStatuses/08584187856876786135
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-7kzpgt/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:276]: failed to create cluster customer resources for node-limit test
Unexpected error:
    <*fmt.wrapError | 0xc000b1a640>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-hxvbhg" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-hxvbhg/operationStatuses/08584187856876786135
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-7kzpgt/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-08' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:85]: failed to create customer resources for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f84de0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-tls-endpoint-hcp-cluster-nkdnjx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-nkdnjx/operationStatuses/08584187858040367930
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-vbsggw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oo7pyvaeowk44\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:85]: failed to create customer resources for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f84de0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-tls-endpoint-hcp-cluster-nkdnjx" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-tls-endpoint-hcp-cluster-nkdnjx/operationStatuses/08584187858040367930
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-vbsggw/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oo7pyvaeowk44\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-12' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: FailedIdentityOperation; provider Microsoft.ManagedIdentity

11:44:43 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:132]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000a623c0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-dcqfgd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: FailedIdentityOperation
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "FailedIdentityOperation",
        "message": "Identity operation for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-dcqfgd/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/delayed-rbac-cluster' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/aro-hcp-msi-container-int-shard0-00-13/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager' was not found.\"}}'.'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:132]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000a623c0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-dcqfgd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: FailedIdentityOperation
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "FailedIdentityOperation",
        "message": "Identity operation for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-dcqfgd/providers/Microsoft.RedHatOpenShift/HCPOpenShiftClusters/delayed-rbac-cluster' failed with error 'Failed to perform resource identity operation. Status: 'BadRequest'. Response: '{\"error\":{\"code\":\"BadRequest\",\"message\":\"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/aro-hcp-msi-container-int-shard0-00-13/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager' was not found.\"}}'.'."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:81]: failed to create customer resources with private key vault
Unexpected error:
    <*fmt.wrapError | 0xc000c04500>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cilium-cluster-bknd4f" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cilium-cluster-bknd4f/operationStatuses/08584187857020344075
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4gslc2/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-4vnk4izirdfzu\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:81]: failed to create customer resources with private key vault
Unexpected error:
    <*fmt.wrapError | 0xc000c04500>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cilium-cluster-bknd4f" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cilium-cluster-bknd4f/operationStatuses/08584187857020344075
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4gslc2/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-4vnk4izirdfzu\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-03' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: DeploymentFailed; detail code ResourceGroupNotFound; detail message resource group '<resource-group>' could not be found.

11:44:43 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:87]: failed to create customer resources for admin credential lifecycle cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f9ec40>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-admin-cred-lifecycle-pnnk7t-pb8dz9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-pnnk7t-pb8dz9/operationStatuses/08584187858025314704
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-9hbvc5/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-es7labf5uzmr2\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:87]: failed to create customer resources for admin credential lifecycle cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f9ec40>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-admin-cred-lifecycle-pnnk7t-pb8dz9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-admin-cred-lifecycle-pnnk7t-pb8dz9/operationStatuses/08584187858025314704
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-9hbvc5/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-es7labf5uzmr2\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          },\r\n          {\r\n            \"code\": \"ResourceGroupNotFound\",\r\n            \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "NotFound",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"ResourceGroupNotFound\",\r\n    \"message\": \"Resource group 'aro-hcp-msi-container-int-shard0-00-14' could not be found.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

Environment: INT