CIHealth Run Log

Environment: DEV

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
22:40:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 33c7d53 · merge 31af859	n/a	provision	failed	★#4041 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid 22:40:36 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph Show raw failure time=2026-02-23T23:07:46.019Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-02-23T23:07:48.372Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra time=2026-02-23T23:07:49.508Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=2f69a9b3d3b3969b6727c934e5aa9eb4b4597b1497c235205c0981d981c1d45b time=2026-02-23T23:09:20.406Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j6603264/providers/Microsoft.Resources/deployments/2f69a9b3d3b3969b6727c934e5aa9eb4b4597b1497c235205c0981d981c1d45b/operationStatuses/08584297188166133511\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6603264/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6603264/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j6603264\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 0e5b97a3-ffae-4ca3-bdc8-de32bea2ded3:2/23/2026 11:08:54 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
22:24:52	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6881981	n/a	e2e	failed	⚠#4171 (open)	11	e2e (11) ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 22:24:52 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:92]: Unexpected error: <fmt.wrapError \| 0xc000199000>: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-stz27g" to finish getting creds: GET https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/c6979eba-31d6-4230-aec4-396b24b8277d -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for hcpCluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-stz27g\" to finish getting creds: GET https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/c6979eba-31d6-4230-aec4-396b24b8277d\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000bf64e0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "eb1ebdf1-825a-48e9-bf00-f8e13c300546", ], "Date": [ "Mon, 23 Feb 2026 23:40:38 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["3713"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000198c60>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j5426048.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/c6979eba-31d6-4230-aec4-396b24b8277d", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T23:40:09Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Authorization": ["redacted"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:92]: Unexpected error: <fmt.wrapError \| 0xc000199000>: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-stz27g" to finish getting creds: GET https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/c6979eba-31d6-4230-aec4-396b24b8277d -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for hcpCluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-stz27g\" to finish getting creds: GET https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/c6979eba-31d6-4230-aec4-396b24b8277d\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000bf64e0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "eb1ebdf1-825a-48e9-bf00-f8e13c300546", ], "Date": [ "Mon, 23 Feb 2026 23:40:38 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["3713"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000198c60>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j5426048.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/c6979eba-31d6-4230-aec4-396b24b8277d", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T23:40:09Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Authorization": ["redacted"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred Interrupted by User 22:24:52 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:51]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:51]: Interrupted by User CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded 22:24:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error: <fmt.wrapErrors \| 0xc00091e960>: failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-b5mk8s, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-b5mk8s" to finish creating: context deadline exceeded { msg: "failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-b5mk8s, error: failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-b5mk8s\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc000f28f30>{ s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-b5mk8s", }, <fmt.wrapError \| 0xc000b0a420>{ msg: "failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-b5mk8s\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:94]: Unexpected error: <fmt.wrapErrors \| 0xc00091e960>: failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-b5mk8s, error: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-b5mk8s" to finish creating: context deadline exceeded { msg: "failed to create NodePool nodepool-128GiB, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-b5mk8s, error: failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-b5mk8s\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc000f28f30>{ s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nodepool-128GiB in resource group clusternp128-b5mk8s", }, <fmt.wrapError \| 0xc000b0a420>{ msg: "failed waiting for nodepool=\"nodepool-128GiB\" for cluster \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-b5mk8s\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded 22:24:52 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:120]: Unexpected error: <fmt.wrapErrors \| 0xc0009f4ae0>: failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-rm6sj8, error: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-rm6sj8" to finish creating: context deadline exceeded { msg: "failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-rm6sj8, error: failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-rm6sj8\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc0005e8080>{ s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-rm6sj8", }, <fmt.wrapError \| 0xc0003da820>{ msg: "failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-rm6sj8\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:120]: Unexpected error: <fmt.wrapErrors \| 0xc0009f4ae0>: failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-rm6sj8, error: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-rm6sj8" to finish creating: context deadline exceeded { msg: "failed to create NodePool ea-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-rm6sj8, error: failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-rm6sj8\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc0005e8080>{ s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool ea-np-1 in resource group external-auth-cluster-rm6sj8", }, <fmt.wrapError \| 0xc0003da820>{ msg: "failed waiting for nodepool=\"ea-np-1\" for cluster \"ea-cluster\" in resourcegroup=\"external-auth-cluster-rm6sj8\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded 22:24:52 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error: <fmt.wrapErrors \| 0xc0011a6960>: failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-ql6v57, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-ql6v57" to finish creating: context deadline exceeded { msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-ql6v57, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ql6v57\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc0010ebbb0>{ s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-ql6v57", }, <fmt.wrapError \| 0xc000b7c300>{ msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ql6v57\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: Unexpected error: <fmt.wrapErrors \| 0xc0011a6960>: failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-ql6v57, error: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-ql6v57" to finish creating: context deadline exceeded { msg: "failed to create NodePool arm64-vm-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-ql6v57, error: failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ql6v57\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc0010ebbb0>{ s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool arm64-vm-np-1 in resource group arm64-vm-cluster-ql6v57", }, <fmt.wrapError \| 0xc000b7c300>{ msg: "failed waiting for nodepool=\"arm64-vm-np-1\" for cluster \"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-ql6v57\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded 22:24:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc0009593e0>: failed to create NodePool np-1, caused by: timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-k9hw2x, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-k9hw2x" to finish creating: context deadline exceeded { msg: "failed to create NodePool np-1, caused by: timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-k9hw2x, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-k9hw2x\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc001021300>{ s: "timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-k9hw2x", }, <fmt.wrapError \| 0xc000486ac0>{ msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-k9hw2x\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc0009593e0>: failed to create NodePool np-1, caused by: timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-k9hw2x, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-k9hw2x" to finish creating: context deadline exceeded { msg: "failed to create NodePool np-1, caused by: timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-k9hw2x, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-k9hw2x\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc001021300>{ s: "timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-k9hw2x", }, <fmt.wrapError \| 0xc000486ac0>{ msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-k9hw2x\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded 22:24:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error: <fmt.wrapErrors \| 0xc000bf0390>: failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-2xnmlr, error: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-2xnmlr" to finish creating: context deadline exceeded { msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-2xnmlr, error: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-2xnmlr\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc0006fa270>{ s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-2xnmlr", }, <fmt.wrapError \| 0xc00040a200>{ msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-2xnmlr\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error: <fmt.wrapErrors \| 0xc000bf0390>: failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-2xnmlr, error: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-2xnmlr" to finish creating: context deadline exceeded { msg: "failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-2xnmlr, error: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-2xnmlr\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc0006fa270>{ s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group basic-cluster-2xnmlr", }, <fmt.wrapError \| 0xc00040a200>{ msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-2xnmlr\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 22:24:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:156]: Unexpected error: <fmt.wrapError \| 0xc0002f0320>: failed to start credential request: POST https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-s94mmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-s94mmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000908780>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "f1726be7-59f9-42d1-9146-2bd34299320f", ], "Date": [ "Mon, 23 Feb 2026 23:42:37 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["450"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc0002f0200>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j5426048.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-s94mmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T23:42:25Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Accept": ["application/json"], "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "Authori... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:156]: Unexpected error: <fmt.wrapError \| 0xc0002f0320>: failed to start credential request: POST https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-s94mmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-s94mmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000908780>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "f1726be7-59f9-42d1-9146-2bd34299320f", ], "Date": [ "Mon, 23 Feb 2026 23:42:37 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["450"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc0002f0200>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j5426048.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-s94mmw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T23:42:25Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Accept": ["application/json"], "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "Authori... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 22:24:52 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapError \| 0xc0008287a0>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-ph7rvf" to finish getting creds: GET https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/a03227c3-c2cf-4efe-b8f2-9726e7f24621 -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-ph7rvf\" to finish getting creds: GET https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/a03227c3-c2cf-4efe-b8f2-9726e7f24621\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000d88e40>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "744d8a96-bb52-44f4-af79-55142559dffb", ], "Date": [ "Mon, 23 Feb 2026 23:40:35 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["3943"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], }, Body: <exported.nopClosingBytesReader \| 0xc0008286e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j5426048.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/a03227c3-c2cf-4efe-b8f2-9726e7f24621", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T23:40:05Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Authorization": ["redacted"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error: <fmt.wrapError \| 0xc0008287a0>: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-ph7rvf" to finish getting creds: GET https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/a03227c3-c2cf-4efe-b8f2-9726e7f24621 -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-ph7rvf\" to finish getting creds: GET https://rp.j5426048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/a03227c3-c2cf-4efe-b8f2-9726e7f24621\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000d88e40>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "744d8a96-bb52-44f4-af79-55142559dffb", ], "Date": [ "Mon, 23 Feb 2026 23:40:35 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["3943"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], }, Body: <exported.nopClosingBytesReader \| 0xc0008286e0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j5426048.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/a03227c3-c2cf-4efe-b8f2-9726e7f24621", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T23:40:05Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Authorization": ["redacted"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred Interrupted by User 22:24:52 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User timeout during CreateHCPClusterAndWait; context deadline exceeded 22:24:52 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error: <fmt.wrapErrors \| 0xc0006e6960>: failed to create HCP cluster list-test-cluster-gndgt6, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5, error: failed waiting for cluster="list-test-cluster-gndgt6" in resourcegroup="cluster-listing-z2gqj5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5, error: context deadline exceeded { msg: "failed to create HCP cluster list-test-cluster-gndgt6, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5, error: failed waiting for cluster=\"list-test-cluster-gndgt6\" in resourcegroup=\"cluster-listing-z2gqj5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000b77d90>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5", }, <fmt.wrapErrors \| 0xc0006e6930>{ msg: "failed waiting for cluster=\"list-test-cluster-gndgt6\" in resourcegroup=\"cluster-listing-z2gqj5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000b77d90>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error: <fmt.wrapErrors \| 0xc0006e6960>: failed to create HCP cluster list-test-cluster-gndgt6, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5, error: failed waiting for cluster="list-test-cluster-gndgt6" in resourcegroup="cluster-listing-z2gqj5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5, error: context deadline exceeded { msg: "failed to create HCP cluster list-test-cluster-gndgt6, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5, error: failed waiting for cluster=\"list-test-cluster-gndgt6\" in resourcegroup=\"cluster-listing-z2gqj5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000b77d90>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5", }, <fmt.wrapErrors \| 0xc0006e6930>{ msg: "failed waiting for cluster=\"list-test-cluster-gndgt6\" in resourcegroup=\"cluster-listing-z2gqj5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000b77d90>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-gndgt6 in resource group cluster-listing-z2gqj5", }, <context.deadlineExceededError>{}, ], }, ], } occurred
21:49:51	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head a7ade8f · merge ce78feb	n/a	provision	failed	⚠#4184 (merged)	1	provision (1) failed to apply namespace multicluster-engine: Unauthorized 21:49:51 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce · suite step graph Show raw failure time=2026-02-23T22:36:07.107Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce description="Step deploy-mce\n Kind: Helm\n" time=2026-02-23T22:36:07.476Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f imageRegistry:arohcpsvcdev.azurecr.io imageRootRepository:acm-d-cache]" time=2026-02-23T22:36:07.476Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce time=2026-02-23T22:36:07.941Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace multicluster-engine: Unauthorized"
21:37:40	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 1a8a9e6 · merge 1a8a9e6	n/a	provision	failed	★#4111 (merged)	1	provision (1) failed pre-install: Unauthorized 21:37:40 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/prometheus · suite step graph Show raw failure time=2026-02-23T22:19:28.846Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-02-23T22:19:29.685Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpdev kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:c3621782e405e59b40e1b64e8b8722855f5c348a9b6fe3ac58c54ad1a34afbb9]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen tag:v1.5.2]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:f97255b7629052f03e21d16171923e7095a9608526196af59e2989a1eb01af9c] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:f9a0f85d9a348f7f83e4b1fcd602eb12fcee0c83b0f7e4eb77d205f6af4311c9]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:3927b209-ebf6-410a-83ed-3dff0dba2908]] prometheusSpec:map[externalLabels:map[cluster:prow-j9791104-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j9791104-mgmt-1-zmnv.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-ddb14f38136b41b1af43306708ab5bb9/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:375d727bf8137d574b5aad4f94881fade0a57556ddf52a3f98322e5d561286af] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j9791104-mgmt-1-zmnv.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-1be76b6a777f4b70aaebec7649d69aa6/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-02-23T22:19:29.685Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-02-23T22:19:29.980Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus namespace=prometheus time=2026-02-23T22:19:29.980Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus dryRun=false time=2026-02-23T22:19:29.980Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-02-23T22:19:30.259Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-02-23T22:21:15.269Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="failed pre-install: Unauthorized" time=2026-02-23T22:21:15.269Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-02-23T22:21:15.269Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-02-23T22:21:22.605Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus release=arohcp-monitor namespace=prometheus status=failed description="Release \"arohcp-monitor\" failed: failed pre-install: Unauthorized" values="map[environment:arohcpdev kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:c3621782e405e59b40e1b64e8b8722855f5c348a9b6fe3ac58c54ad1a34afbb9]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen tag:v1.5.2]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:f97255b7629052f03e21d16171923e7095a9608526196af59e2989a1eb01af9c] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:f9a0f85d9a348f7f83e4b1fcd602eb12fcee0c83b0f7e4eb77d205f6af4311c9]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:3927b209-ebf6-410a-83ed-3dff0dba2908]] prometheusSpec:map[externalLabels:map[cluster:prow-j9791104-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j9791104-mgmt-1-zmnv.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-ddb14f38136b41b1af43306708ab5bb9/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:375d727bf8137d574b5aad4f94881fade0a57556ddf52a3f98322e5d561286af] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j9791104-mgmt-1-zmnv.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-1be76b6a777f4b70aaebec7649d69aa6/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-02-23T22:21:22.605Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-02-23T22:21:22.605Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed pre-install: Unauthorized"
21:12:46	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e1e8f06 · merge e1e8f06	n/a	provision	failed	★#4144 (merged)	1	provision (1) unable to continue with install: could not get information about the resource ConfigMap "admission-policies-controller-config" in namespace "<namespace>": Unauthorized 21:12:46 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/acrpull · suite step graph Show raw failure time=2026-02-23T22:05:58.993Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull description="Step acrpull\n Kind: Helm\n" time=2026-02-23T22:05:59.522Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull values="map[image:mcr.microsoft.com/aks/msi-acrpull@sha256:042511defb0842f02593e48e68d83a20a1af4265db4e23fca484356e34a6c883 namespace:acrpull replicas:2 ttlRotationFraction:0.5]" time=2026-02-23T22:05:59.522Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-02-23T22:05:59.858Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull namespace=acrpull time=2026-02-23T22:05:59.858Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull dryRun=false time=2026-02-23T22:05:59.858Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-02-23T22:06:00.119Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-02-23T22:06:01.512Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull err="unable to continue with install: could not get information about the resource ConfigMap \"admission-policies-controller-config\" in namespace \"acrpull\": Unauthorized" time=2026-02-23T22:06:01.512Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-02-23T22:06:01.512Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-02-23T22:06:01.627Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull err="failed to get status for release acrpull: release: not found" time=2026-02-23T22:06:01.627Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull time=2026-02-23T22:06:01.628Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=acrpull err="error running Helm release deployment Step, failed to deploy helm release: unable to continue with install: could not get information about the resource ConfigMap \"admission-policies-controller-config\" in namespace \"acrpull\": Unauthorized"
21:01:20	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head e41fc5c · merge 26b2f21	n/a	e2e	failed	★#4062 (merged)	1	e2e (1) cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) 21:01:20 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc0005723f0>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc0005723f0>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred
20:59:44	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6881981	n/a	unknown	failed	#4171 (open)	1	non-artifact-backed failure (no junit artifacts)
20:58:10	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head cfdce34 · merge 3ca804d	n/a	n/a	passed	#3679 (merged)	0	n/a
20:40:09	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 1a8a9e6 · merge 1a8a9e6	n/a	provision	failed	★#4111 (merged)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 20:40:09 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-02-23T21:05:03.327Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-02-23T21:05:13.449Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=4b0d3e19-2165-4223-97a8-ff2f9ade0032 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=4d92d853-4642-4a0f-b999-19cdd2f0e657 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj3347200.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj3347200.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j3347200 -n arohcp-prow-dbcs-j3347200 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3347200/providers/Microsoft.Network/dnszones/j3347200.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j3347200-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j3347200-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpdev \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set global.imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j3347200-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"45be3786-5508-4596-98f9-f0fb68642951\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j3347200-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3347200-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j3347200-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nerror: You must be logged in to the server (error when creating \"STDIN\": Unauthorized)\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
20:22:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1c91345	n/a	provision	failed	★#4133 (open)	1	provision (1) failed post-install: Unauthorized 20:22:59 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph Show raw failure time=2026-02-23T21:19:55.937Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-02-23T21:19:56.533Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-02-23T21:19:56.533Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T21:19:56.601Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-02-23T21:19:56.601Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-02-23T21:19:56.601Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T21:19:56.725Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T21:21:46.712Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: Unauthorized" time=2026-02-23T21:21:46.712Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T21:21:46.712Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T21:21:46.868Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed to get status for release mce-config: query: failed to query with labels: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/multicluster-engine/secrets?labelSelector=name%3Dmce-config%2Cowner%3Dhelm\\\": Post \\\"https://guard.699cc16bf7e86200018b5766.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.699cc16bf7e86200018b5766.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (get secrets)" time=2026-02-23T21:21:46.868Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T21:21:46.868Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
19:51:05	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 33c7d53 · merge 31af859	n/a	e2e	failed	★#4041 (merged)	2	e2e (2) ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 19:51:05 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:152]: Unexpected error: <fmt.wrapError \| 0xc000684120>: failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-r9wsgm" to finish updating: GET https://rp.j4880128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/023741af-39c7-45a1-a4ff-40c0ae8371a7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/023741af-39c7-45a1-a4ff-40c0ae8371a7", "name": "023741af-39c7-45a1-a4ff-40c0ae8371a7", "status": "Failed", "startTime": "2026-02-23T21:12:03.035773476Z", "endTime": "2026-02-23T21:12:15.190412435Z", "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-r9wsgm\" to finish updating: GET https://rp.j4880128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/023741af-39c7-45a1-a4ff-40c0ae8371a7\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/023741af-39c7-45a1-a4ff-40c0ae8371a7\",\n \"name\": \"023741af-39c7-45a1-a4ff-40c0ae8371a7\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T21:12:03.035773476Z\",\n \"endTime\": \"2026-02-23T21:12:15.190412435Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00073a510>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["18"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "c0a04a77-501c-4436-9dc7-7ae38d30e33d", ], "Date": [ "Mon, 23 Feb 2026 21:12:23 GMT", ], "Content-Length": ["460"], }, Body: <exported.nopClosingBytesReader \| 0xc000684000>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 48, 50, 51, 55, 52, 49, 97, 102, 45, 51, 57, 99, 55, 45, 52, 53, 97, 49, 45, 97, 52, 102, 102, 45, 52, 48, 99, 48, 97, 101, 56, 51, 55, 49, 97, 55, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 48, 50, 51, 55, 52, 49, 97, 102, 45, 51, 57, 99, 55, 45, 52, 53, 97, 49, 45, 97, 52, 102, 102, 45, 52, 48, 99, 48, 97, 101, 56, 51, 55, 49, 97, 55, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 50, 45, 50, 51, 84, 50, 49, 58, 49, 50, 58, 48, 51, 46, 48, 51, 53, 55, 55, 51, 52, 55, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 50, 45, 50, 51, 84, 50, 49, 58, 49, 50, 58, 49, 53, 46, 49, 57, 48, 52, 49, 50, 52, 51, 53, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 1... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:152]: Unexpected error: <fmt.wrapError \| 0xc000684120>: failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-r9wsgm" to finish updating: GET https://rp.j4880128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/023741af-39c7-45a1-a4ff-40c0ae8371a7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/023741af-39c7-45a1-a4ff-40c0ae8371a7", "name": "023741af-39c7-45a1-a4ff-40c0ae8371a7", "status": "Failed", "startTime": "2026-02-23T21:12:03.035773476Z", "endTime": "2026-02-23T21:12:15.190412435Z", "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-r9wsgm\" to finish updating: GET https://rp.j4880128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/023741af-39c7-45a1-a4ff-40c0ae8371a7\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/023741af-39c7-45a1-a4ff-40c0ae8371a7\",\n \"name\": \"023741af-39c7-45a1-a4ff-40c0ae8371a7\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T21:12:03.035773476Z\",\n \"endTime\": \"2026-02-23T21:12:15.190412435Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00073a510>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["18"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "c0a04a77-501c-4436-9dc7-7ae38d30e33d", ], "Date": [ "Mon, 23 Feb 2026 21:12:23 GMT", ], "Content-Length": ["460"], }, Body: <exported.nopClosingBytesReader \| 0xc000684000>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 48, 50, 51, 55, 52, 49, 97, 102, 45, 51, 57, 99, 55, 45, 52, 53, 97, 49, 45, 97, 52, 102, 102, 45, 52, 48, 99, 48, 97, 101, 56, 51, 55, 49, 97, 55, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 48, 50, 51, 55, 52, 49, 97, 102, 45, 51, 57, 99, 55, 45, 52, 53, 97, 49, 45, 97, 52, 102, 102, 45, 52, 48, 99, 48, 97, 101, 56, 51, 55, 49, 97, 55, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 50, 45, 50, 51, 84, 50, 49, 58, 49, 50, 58, 48, 51, 46, 48, 51, 53, 55, 55, 51, 52, 55, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 50, 45, 50, 51, 84, 50, 49, 58, 49, 50, 58, 49, 53, 46, 49, 57, 48, 52, 49, 50, 52, 51, 53, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 1... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 19:51:05 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:400]: Unexpected error: <exported.ResponseError \| 0xc000d04180>: GET https://rp.j4880128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-q8fgkx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["2517"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "ac33f72e-d3e4-4db6-8cf5-e2bf14a7ce0c", ], "Date": [ "Mon, 23 Feb 2026 21:11:22 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc000410500>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j4880128.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-q8fgkx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Accept": ["application/json"], "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T21:11:01Z\"}", ], "Authorization": ["redacted"], }, Body: nil, GetBody: nil, ContentLength: 0, TransferEncoding: nil, Close: false, Host: "rp.j4880128.hcpsvc.osadev.cloud", Form: nil, PostForm: nil, MultipartForm: nil, Trailer: nil, RemoteAddr: "", RequestURI: "", TLS: nil, Cancel: nil, Response: nil, Pattern: "", ctx: <suppressed context>, pat: nil, matches: nil, otherValues: nil, }, TLS: { Version: 772, HandshakeComplete: true, DidResume: false, CipherSuite: 4865, CurveID: 29, NegotiatedProtocol: "h2", NegotiatedProtocolIsMutual: true, ServerName: "rp.j4880128.hcpsvc.osadev.cl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:400]: Unexpected error: <exported.ResponseError \| 0xc000d04180>: GET https://rp.j4880128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-q8fgkx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["2517"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "ac33f72e-d3e4-4db6-8cf5-e2bf14a7ce0c", ], "Date": [ "Mon, 23 Feb 2026 21:11:22 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc000410500>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j4880128.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-q8fgkx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Accept": ["application/json"], "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T21:11:01Z\"}", ], "Authorization": ["redacted"], }, Body: nil, GetBody: nil, ContentLength: 0, TransferEncoding: nil, Close: false, Host: "rp.j4880128.hcpsvc.osadev.cloud", Form: nil, PostForm: nil, MultipartForm: nil, Trailer: nil, RemoteAddr: "", RequestURI: "", TLS: nil, Cancel: nil, Response: nil, Pattern: "", ctx: <suppressed context>, pat: nil, matches: nil, otherValues: nil, }, TLS: { Version: 772, HandshakeComplete: true, DidResume: false, CipherSuite: 4865, CurveID: 29, NegotiatedProtocol: "h2", NegotiatedProtocolIsMutual: true, ServerName: "rp.j4880128.hcpsvc.osadev.cl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
19:45:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head e41fc5c · merge 26b2f21	n/a	provision	failed	⚠#4062 (merged)	1	provision (1) failed to install CRD crds/apps.open-cluster-management.io_placementrules_crd_v1.yaml: Unauthorized 19:45:36 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph Show raw failure time=2026-02-23T20:39:57.079Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-02-23T20:39:57.597Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-02-23T20:39:57.597Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T20:39:57.656Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-02-23T20:39:57.656Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-02-23T20:39:57.656Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T20:39:57.770Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T20:39:58.614Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed to install CRD crds/apps.open-cluster-management.io_placementrules_crd_v1.yaml: Unauthorized" time=2026-02-23T20:39:58.614Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T20:39:58.614Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T20:39:58.726Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed to get status for release mce-config: release: not found" time=2026-02-23T20:39:58.726Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T20:39:58.727Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed to install CRD crds/apps.open-cluster-management.io_placementrules_crd_v1.yaml: Unauthorized"
19:43:44	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6881981	n/a	provision	failed	⚠#4171 (open)	1	provision (1) failed to apply namespace multicluster-engine: an error on the server ("Internal Server Error: \\"/api/v1/namespaces/multicluster-engine?fieldManager=templatize&force=false\\": Post \\"<url>": unexpected EOF") has prevented the request from succeeding 19:43:44 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-crds · suite step graph Show raw failure time=2026-02-23T20:05:11.221Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds description="Step deploy-mce-crds\n Kind: Helm\n" time=2026-02-23T20:05:12.058Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds values=map[] time=2026-02-23T20:05:12.058Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds time=2026-02-23T20:05:12.365Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-crds err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace multicluster-engine: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/multicluster-engine?fieldManager=templatize&force=false\\\": Post \\\"https://guard.699caf829830e200014ad01e.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": unexpected EOF\") has prevented the request from succeeding (patch namespaces multicluster-engine)"
19:09:53	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6881981	n/a	provision	failed	⚠#4171 (open)	1	provision (1) unable to continue with install: could not get information about the resource ValidatingAdmissionPolicy "controller-secret-mutation-policy" in namespace "": Unauthorized 19:09:53 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/acrpull · suite step graph Show raw failure time=2026-02-23T19:33:49.440Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull description="Step acrpull\n Kind: Helm\n" time=2026-02-23T19:33:49.988Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull values="map[image:mcr.microsoft.com/aks/msi-acrpull@sha256:042511defb0842f02593e48e68d83a20a1af4265db4e23fca484356e34a6c883 namespace:acrpull replicas:2 ttlRotationFraction:0.5]" time=2026-02-23T19:33:49.988Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-02-23T19:33:50.243Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull namespace=acrpull time=2026-02-23T19:33:50.243Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull dryRun=false time=2026-02-23T19:33:50.243Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-02-23T19:33:50.529Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-02-23T19:33:54.878Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull err="unable to continue with install: could not get information about the resource ValidatingAdmissionPolicy \"controller-secret-mutation-policy\" in namespace \"\": Unauthorized" time=2026-02-23T19:33:54.878Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-02-23T19:33:54.878Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-02-23T19:33:55.047Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull err="failed to get status for release acrpull: release: not found" time=2026-02-23T19:33:55.047Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull time=2026-02-23T19:33:55.047Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=acrpull err="error running Helm release deployment Step, failed to deploy helm release: unable to continue with install: could not get information about the resource ValidatingAdmissionPolicy \"controller-secret-mutation-policy\" in namespace \"\": Unauthorized"
18:55:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head e887b94 · merge 7178883	n/a	e2e	failed	★#4183 (merged)	4	e2e (4) ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 18:55:36 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error: <errors.joinError \| 0xc000d92e88>: failed to cleanup resource group: failed listing hcp clusters in resourcegroup="rg-cluster-back-version-w2lgmz": GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-w2lgmz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <fmt.wrapError \| 0xc000a850e0>{ msg: "failed to cleanup resource group: failed listing hcp clusters in resourcegroup=\"rg-cluster-back-version-w2lgmz\": GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-w2lgmz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a850c0>{ msg: "failed listing hcp clusters in resourcegroup=\"rg-cluster-back-version-w2lgmz\": GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-w2lgmz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000bb55c0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "15cf8b9f-7a70-4688-8d1f-e33f58872fbf", ], "Date": [ "Mon, 23 Feb 2026 20:24:59 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["3326"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000a85000>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error: <errors.joinError \| 0xc000d92e88>: failed to cleanup resource group: failed listing hcp clusters in resourcegroup="rg-cluster-back-version-w2lgmz": GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-w2lgmz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <fmt.wrapError \| 0xc000a850e0>{ msg: "failed to cleanup resource group: failed listing hcp clusters in resourcegroup=\"rg-cluster-back-version-w2lgmz\": GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-w2lgmz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a850c0>{ msg: "failed listing hcp clusters in resourcegroup=\"rg-cluster-back-version-w2lgmz\": GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-w2lgmz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000bb55c0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "15cf8b9f-7a70-4688-8d1f-e33f58872fbf", ], "Date": [ "Mon, 23 Feb 2026 20:24:59 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["3326"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000a85000>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 18:55:36 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:122]: Unexpected error: <fmt.wrapError \| 0xc00040a8e0>: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-kms5gv" to finish getting creds: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/dacd8331-775a-4c12-a716-feb199663ffa -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kms5gv\" to finish getting creds: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/dacd8331-775a-4c12-a716-feb199663ffa\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000806a80>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["3275"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "5588a858-aa9f-4ebb-a3ae-9b65d45e0b64", ], "Date": [ "Mon, 23 Feb 2026 20:21:01 GMT", ], "Content-Length": ["107"], }, Body: <exported.nopClosingBytesReader \| 0xc00040a820>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j7383168.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/dacd8331-775a-4c12-a716-feb199663ffa", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T20:20:39Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "Authorization": ["redacted"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:122]: Unexpected error: <fmt.wrapError \| 0xc00040a8e0>: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-kms5gv" to finish getting creds: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/dacd8331-775a-4c12-a716-feb199663ffa -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-kms5gv\" to finish getting creds: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/dacd8331-775a-4c12-a716-feb199663ffa\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000806a80>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["3275"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "5588a858-aa9f-4ebb-a3ae-9b65d45e0b64", ], "Date": [ "Mon, 23 Feb 2026 20:21:01 GMT", ], "Content-Length": ["107"], }, Body: <exported.nopClosingBytesReader \| 0xc00040a820>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j7383168.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationResults/dacd8331-775a-4c12-a716-feb199663ffa", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T20:20:39Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ], "Authorization": ["redacted"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 18:55:36 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:179]: Unexpected error: <fmt.wrapError \| 0xc0003312c0>: failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-rw5g7c" to finish updating: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf2cef3-df89-4589-ac68-2eea4c7a891f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf2cef3-df89-4589-ac68-2eea4c7a891f", "name": "9bf2cef3-df89-4589-ac68-2eea4c7a891f", "status": "Failed", "startTime": "2026-02-23T20:34:06.807375973Z", "endTime": "2026-02-23T20:34:18.455374606Z", "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-rw5g7c\" to finish updating: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf2cef3-df89-4589-ac68-2eea4c7a891f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf2cef3-df89-4589-ac68-2eea4c7a891f\",\n \"name\": \"9bf2cef3-df89-4589-ac68-2eea4c7a891f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T20:34:06.807375973Z\",\n \"endTime\": \"2026-02-23T20:34:18.455374606Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0008e8cc0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["460"], "X-Envoy-Upstream-Service-Time": ["12"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "cf1fd5bc-43de-4923-b058-b3b258ead83a", ], "Date": [ "Mon, 23 Feb 2026 20:34:27 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc000330d00>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 57, 98, 102, 50, 99, 101, 102, 51, 45, 100, 102, 56, 57, 45, 52, 53, 56, 57, 45, 97, 99, 54, 56, 45, 50, 101, 101, 97, 52, 99, 55, 97, 56, 57, 49, 102, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 57, 98, 102, 50, 99, 101, 102, 51, 45, 100, 102, 56, 57, 45, 52, 53, 56, 57, 45, 97, 99, 54, 56, 45, 50, 101, 101, 97, 52, 99, 55, 97, 56, 57, 49, 102, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 50, 45, 50, 51, 84, 50, 48, 58, 51, 52, 58, 48, 54, 46, 56, 48, 55, 51, 55, 53, 57, 55, 51, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 50, 45, 50, 51, 84, 50, 48, 58, 51, 52, 58, 49, 56, 46, 52, 53, 53, 51, 55, 52, 54, 48, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:179]: Unexpected error: <fmt.wrapError \| 0xc0003312c0>: failed waiting for nodepool="np-update-nodes" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-rw5g7c" to finish updating: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf2cef3-df89-4589-ac68-2eea4c7a891f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf2cef3-df89-4589-ac68-2eea4c7a891f", "name": "9bf2cef3-df89-4589-ac68-2eea4c7a891f", "status": "Failed", "startTime": "2026-02-23T20:34:06.807375973Z", "endTime": "2026-02-23T20:34:18.455374606Z", "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed waiting for nodepool=\"np-update-nodes\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-rw5g7c\" to finish updating: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf2cef3-df89-4589-ac68-2eea4c7a891f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf2cef3-df89-4589-ac68-2eea4c7a891f\",\n \"name\": \"9bf2cef3-df89-4589-ac68-2eea4c7a891f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T20:34:06.807375973Z\",\n \"endTime\": \"2026-02-23T20:34:18.455374606Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0008e8cc0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["460"], "X-Envoy-Upstream-Service-Time": ["12"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "cf1fd5bc-43de-4923-b058-b3b258ead83a", ], "Date": [ "Mon, 23 Feb 2026 20:34:27 GMT", ], }, Body: <exported.nopClosingBytesReader \| 0xc000330d00>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 57, 98, 102, 50, 99, 101, 102, 51, 45, 100, 102, 56, 57, 45, 52, 53, 56, 57, 45, 97, 99, 54, 56, 45, 50, 101, 101, 97, 52, 99, 55, 97, 56, 57, 49, 102, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 57, 98, 102, 50, 99, 101, 102, 51, 45, 100, 102, 56, 57, 45, 52, 53, 56, 57, 45, 97, 99, 54, 56, 45, 50, 101, 101, 97, 52, 99, 55, 97, 56, 57, 49, 102, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 50, 45, 50, 51, 84, 50, 48, 58, 51, 52, 58, 48, 54, 46, 56, 48, 55, 51, 55, 53, 57, 55, 51, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 50, 45, 50, 51, 84, 50, 48, 58, 51, 52, 58, 49, 56, 46, 52, 53, 53, 51, 55, 52, 54, 48, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 18:55:36 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error: <fmt.wrapError \| 0xc0005012c0>: failed to create HCP cluster list-test-cluster-8xtkh8: failed waiting for cluster="list-test-cluster-8xtkh8" in resourcegroup="cluster-listing-7cs2h5" to finish creating: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-7cs2h5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-8xtkh8 -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster list-test-cluster-8xtkh8: failed waiting for cluster=\"list-test-cluster-8xtkh8\" in resourcegroup=\"cluster-listing-7cs2h5\" to finish creating: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-7cs2h5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-8xtkh8\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000501280>{ msg: "failed waiting for cluster=\"list-test-cluster-8xtkh8\" in resourcegroup=\"cluster-listing-7cs2h5\" to finish creating: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-7cs2h5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-8xtkh8\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000e05c20>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "c2acb288-1455-4584-8aa2-761bb0312668", ], "Date": [ "Mon, 23 Feb 2026 20:21:01 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["2930"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc0005011a0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j7383168.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-7cs2h5/providers/Microsoft.RedHatOpenS... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error: <fmt.wrapError \| 0xc0005012c0>: failed to create HCP cluster list-test-cluster-8xtkh8: failed waiting for cluster="list-test-cluster-8xtkh8" in resourcegroup="cluster-listing-7cs2h5" to finish creating: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-7cs2h5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-8xtkh8 -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster list-test-cluster-8xtkh8: failed waiting for cluster=\"list-test-cluster-8xtkh8\" in resourcegroup=\"cluster-listing-7cs2h5\" to finish creating: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-7cs2h5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-8xtkh8\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000501280>{ msg: "failed waiting for cluster=\"list-test-cluster-8xtkh8\" in resourcegroup=\"cluster-listing-7cs2h5\" to finish creating: GET https://rp.j7383168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-7cs2h5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-8xtkh8\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000e05c20>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "c2acb288-1455-4584-8aa2-761bb0312668", ], "Date": [ "Mon, 23 Feb 2026 20:21:01 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["2930"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc0005011a0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j7383168.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-7cs2h5/providers/Microsoft.RedHatOpenS... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
18:36:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 6ab1b69 · merge e43ebba	n/a	unknown	failed	#4180 (merged)	1	non-artifact-backed failure (no junit artifacts)
17:56:28	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head cfdce34 · merge 3ca804d	n/a	provision	failed	★#3679 (merged)	1	provision (1) failed post-install: Unauthorized 17:56:28 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph Show raw failure time=2026-02-23T19:18:18.127Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n" time=2026-02-23T19:18:18.681Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-02-23T19:18:18.681Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T19:18:18.746Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine time=2026-02-23T19:18:18.746Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false time=2026-02-23T19:18:18.746Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T19:18:18.871Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T19:20:08.330Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: Unauthorized" time=2026-02-23T19:20:08.330Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T19:20:08.331Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T19:20:15.915Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config release=mce-config namespace=multicluster-engine status=failed description="Release \"mce-config\" failed: failed post-install: Unauthorized" values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]" time=2026-02-23T19:20:15.917Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config url="https://hcp-dev-us.westus3.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yUQW/bPgzF7/4UhE8NYBeN//gPnYceum6nYhuw7tRhB1liHDUyaVBKggD58IPiJBa2LgmGXZ/5fnpkGGYOAwh6XopGD3dgVFBBNQ6vuHlBHR4tmdoHsdQWMEifVYdHCVdIO8X3Sh/kyfcMACB/QllZjfda85JCXkDeii57dlZvSmUMU+lVlLulC1a7pQ8oJVJrCfPiFOOs7Ss7fG/JWGoPnt/eFnbYjDUnaPfGfKEP2DvedEjhgWlmd54BNGcf0JT6KJ9APQzqJ0WqxQjbsaNpsB8iaqYg7BxK/Hbe3fIKhRRpPBBmojpcsyxSwOA0e84T7iY63/Qofm5noeyO5HKf37/yfpxulLlHOtQl1noxSA7DftRjN+Wq+gtgK7oe2trzhhLZ216hJT/+P015ATeGTQImK5Zu58m1vGC7U/8FlnF7Y+H51i9GJfl74V61KrCcITwenx//Sq3jRrndlLIf77LFskEhDOg/xhPjsy2s5ygIwXbog+p6aDCsEQmujAoY5au8uqnelDdVWf33bfq2nt7W09vnfALX1/Cnmuqmnv7/nE8m2RZe2BIsLJk7S4SSHEYmGO9hegh/vYDZFnrh+HUMWkDPZii+DFKA47aADr1XbUSyGBRoNknzBr3+GQAA//8jnHRHvgUAAA==" time=2026-02-23T19:20:15.919Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config time=2026-02-23T19:20:15.920Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: Unauthorized"
17:52:39	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 33c7d53 · merge 31af859	n/a	provision	failed	★#4041 (merged)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 17:52:39 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-02-23T19:16:58.687Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-02-23T19:19:05.523Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=0ea532e8-59b9-4f3a-accb-19e8b9515289 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=e2711603-879f-4a69-ba0a-7891a926d0a2 && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj6433152.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj6433152.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j6433152 -n arohcp-prow-dbcs-j6433152 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6433152/providers/Microsoft.Network/dnszones/j6433152.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j6433152-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j6433152-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpdev \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set global.imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j6433152-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"cd7c0c88-12e2-40cf-971a-1387b5aa6b5e\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j6433152-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6433152-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j6433152-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Unauthorized\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
17:50:51	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head cfae8a5 · merge e43ebba	n/a	unknown	failed	#4180 (merged)	1	non-artifact-backed failure (no junit artifacts)
17:20:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8f711fa	n/a	e2e	failed	★#4095 (open)	1	e2e (1) cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) 17:20:55 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc000bf3ad0>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc000bf3ad0>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred
17:12:28	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 6b5fd66 · merge 6b5fd66	n/a	provision	failed	★#4015 (merged)	1	provision (1) failed to apply namespace prometheus: Unauthorized 17:12:28 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/prometheus · suite step graph Show raw failure time=2026-02-23T18:33:02.218Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-02-23T18:33:02.914Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus values="map[environment:arohcpdev kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:c3621782e405e59b40e1b64e8b8722855f5c348a9b6fe3ac58c54ad1a34afbb9]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen tag:v1.5.2]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:f97255b7629052f03e21d16171923e7095a9608526196af59e2989a1eb01af9c] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:f9a0f85d9a348f7f83e4b1fcd602eb12fcee0c83b0f7e4eb77d205f6af4311c9]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:42532944-f3c2-4455-b2dd-0a9084a85775]] prometheusSpec:map[externalLabels:map[cluster:prow-j4450304-mgmt-1] hcpRemoteWriteUrl:https://msprom-westus3-prow-j4450304-mgmt-1-aycq.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-beb8c9ee207d4d919bebbce3c0fdda7c/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:375d727bf8137d574b5aad4f94881fade0a57556ddf52a3f98322e5d561286af] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j4450304-mgmt-1-aycq.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-620ba3565e254764a564aac7b9d2be6b/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-02-23T18:33:02.914Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus time=2026-02-23T18:33:03.421Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace prometheus: Unauthorized"
17:09:42	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 1a8a9e6 · merge 1a8a9e6	n/a	e2e	failed	★#4111 (merged)	4	e2e (4) CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded 17:09:42 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc000a284b0>: failed to create NodePool np-1, caused by: timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-75njvv, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-75njvv" to finish creating: context deadline exceeded { msg: "failed to create NodePool np-1, caused by: timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-75njvv, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-75njvv\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc0002012c0>{ s: "timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-75njvv", }, <fmt.wrapError \| 0xc0007821e0>{ msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-75njvv\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: Unexpected error: <fmt.wrapErrors \| 0xc000a284b0>: failed to create NodePool np-1, caused by: timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-75njvv, error: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-75njvv" to finish creating: context deadline exceeded { msg: "failed to create NodePool np-1, caused by: timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-75njvv, error: failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-75njvv\" to finish creating: context deadline exceeded", errs: [ <errors.errorString \| 0xc0002012c0>{ s: "timeout '15.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group pullsecret-test-75njvv", }, <fmt.wrapError \| 0xc0007821e0>{ msg: "failed waiting for nodepool=\"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-75njvv\" to finish creating: context deadline exceeded", err: <context.deadlineExceededError>{}, }, ], } occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 17:09:42 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error: <fmt.wrapError \| 0xc00049aae0>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-5jkqss" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023", "name": "b3b4161b-56c0-44a8-9361-198f0f406023", "status": "Failed", "startTime": "2026-02-23T19:10:28.001047942Z", "endTime": "2026-02-23T19:10:50.761314326Z", "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create NodePool np-1: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5jkqss\" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023\",\n \"name\": \"b3b4161b-56c0-44a8-9361-198f0f406023\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T19:10:28.001047942Z\",\n \"endTime\": \"2026-02-23T19:10:50.761314326Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00049aaa0>{ msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5jkqss\" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023\",\n \"name\": \"b3b4161b-56c0-44a8-9361-198f0f406023\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T19:10:28.001047942Z\",\n \"endTime\": \"2026-02-23T19:10:50.761314326Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ee58c0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "4ae4a3e5-81ec-4625-871a-8dc437860b17", ], "Date": [ "Mon, 23 Feb 2026 19:10:58 GMT", ], "Content-Length": ["460"], "X-Envoy-Upstream-Service-Time": ["8"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc00049a900>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error: <fmt.wrapError \| 0xc00049aae0>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-5jkqss" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023", "name": "b3b4161b-56c0-44a8-9361-198f0f406023", "status": "Failed", "startTime": "2026-02-23T19:10:28.001047942Z", "endTime": "2026-02-23T19:10:50.761314326Z", "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create NodePool np-1: failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5jkqss\" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023\",\n \"name\": \"b3b4161b-56c0-44a8-9361-198f0f406023\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T19:10:28.001047942Z\",\n \"endTime\": \"2026-02-23T19:10:50.761314326Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00049aaa0>{ msg: "failed waiting for nodepool=\"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-5jkqss\" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b3b4161b-56c0-44a8-9361-198f0f406023\",\n \"name\": \"b3b4161b-56c0-44a8-9361-198f0f406023\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T19:10:28.001047942Z\",\n \"endTime\": \"2026-02-23T19:10:50.761314326Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ee58c0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "4ae4a3e5-81ec-4625-871a-8dc437860b17", ], "Date": [ "Mon, 23 Feb 2026 19:10:58 GMT", ], "Content-Length": ["460"], "X-Envoy-Upstream-Service-Time": ["8"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc00049a900>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115,... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 17:09:42 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:156]: Unexpected error: <fmt.wrapError \| 0xc00012baa0>: failed to start credential request: POST https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-rg2wrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-rg2wrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0005baea0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "056b83b5-a55b-48ec-aac1-b1a8fb717f1d", ], "Date": [ "Mon, 23 Feb 2026 19:13:22 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["209"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc00012b9c0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j0502272.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-rg2wrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T19:13:10Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Authorization": ["redacted"], "Accept": ["application/json"], "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:156]: Unexpected error: <fmt.wrapError \| 0xc00012baa0>: failed to start credential request: POST https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-rg2wrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to start credential request: POST https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-rg2wrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0005baea0>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "056b83b5-a55b-48ec-aac1-b1a8fb717f1d", ], "Date": [ "Mon, 23 Feb 2026 19:13:22 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["209"], "Server": ["istio-envoy"], "Content-Type": ["application/json"], }, Body: <exported.nopClosingBytesReader \| 0xc00012b9c0>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "POST", URL: { Scheme: "https", Opaque: "", User: nil, Host: "rp.j0502272.hcpsvc.osadev.cloud", Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-rg2wrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-19/requestAdminCredential", RawPath: "", OmitHost: false, ForceQuery: false, RawQuery: "api-version=2024-06-10-preview", Fragment: "", RawFragment: "", }, Proto: "HTTP/1.1", ProtoMajor: 1, ProtoMinor: 1, Header: { "X-Ms-Arm-Resource-System-Data": [ "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-02-23T19:13:10Z\"}", ], "X-Ms-Identity-Url": [ "https://dummyhost.identity.azure.net", ], "Authorization": ["redacted"], "Accept": ["application/json"], "User-Agent": [ "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)", ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 17:09:42 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: Unexpected error: <fmt.wrapError \| 0xc000614ae0>: failed to create NodePool autoscale-np: failed waiting for nodepool="autoscale-np" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-st6n7x" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b", "name": "850f5057-0f50-4939-9829-71e3e3f7f04b", "status": "Failed", "startTime": "2026-02-23T19:11:21.851714994Z", "endTime": "2026-02-23T19:11:40.778442005Z", "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create NodePool autoscale-np: failed waiting for nodepool=\"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-st6n7x\" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b\",\n \"name\": \"850f5057-0f50-4939-9829-71e3e3f7f04b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T19:11:21.851714994Z\",\n \"endTime\": \"2026-02-23T19:11:40.778442005Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000614900>{ msg: "failed waiting for nodepool=\"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-st6n7x\" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b\",\n \"name\": \"850f5057-0f50-4939-9829-71e3e3f7f04b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T19:11:21.851714994Z\",\n \"endTime\": \"2026-02-23T19:11:40.778442005Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000986cc0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "10f2ae7b-3adc-40f6-ba19-cf8dbb1535ec", ], "Date": [ "Mon, 23 Feb 2026 19:11:42 GMT", ], "Content-Length": ["460"], "X-Envoy-Upstream-Service-Time": ["13"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc0006146e0>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: Unexpected error: <fmt.wrapError \| 0xc000614ae0>: failed to create NodePool autoscale-np: failed waiting for nodepool="autoscale-np" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-st6n7x" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b", "name": "850f5057-0f50-4939-9829-71e3e3f7f04b", "status": "Failed", "startTime": "2026-02-23T19:11:21.851714994Z", "endTime": "2026-02-23T19:11:40.778442005Z", "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { msg: "failed to create NodePool autoscale-np: failed waiting for nodepool=\"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-st6n7x\" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b\",\n \"name\": \"850f5057-0f50-4939-9829-71e3e3f7f04b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T19:11:21.851714994Z\",\n \"endTime\": \"2026-02-23T19:11:40.778442005Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000614900>{ msg: "failed waiting for nodepool=\"autoscale-np\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-st6n7x\" to finish creating: GET https://rp.j0502272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/850f5057-0f50-4939-9829-71e3e3f7f04b\",\n \"name\": \"850f5057-0f50-4939-9829-71e3e3f7f04b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-02-23T19:11:21.851714994Z\",\n \"endTime\": \"2026-02-23T19:11:40.778442005Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000986cc0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "10f2ae7b-3adc-40f6-ba19-cf8dbb1535ec", ], "Date": [ "Mon, 23 Feb 2026 19:11:42 GMT", ], "Content-Length": ["460"], "X-Envoy-Upstream-Service-Time": ["13"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc0006146e0>{ s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
16:52:44	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6881981	n/a	provision	failed	★#4171 (open)	1	provision (1) make[2]: * [Makefile:32: deploy] Error 1 16:52:44 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite step graph Show raw failure time=2026-02-23T17:47:45.426Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Shell\n Command: make deploy\n" time=2026-02-23T17:47:59.291Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Shell Step, failed to execute shell command: make[2]: Entering directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\nkubectl create namespace clusters-service --dry-run=client -o json \| kubectl apply -f - && \\\nIMAGE_PULLER_MI_CLIENT_ID=cd981806-2bc2-46b7-b177-f7edc1738390 && \\\nIMAGE_PULLER_MI_TENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nkubectl label namespace clusters-service \"istio.io/rev=prod-stable\" --overwrite=true && \\\nAZURE_CS_MI_CLIENT_ID=ac5912db-acef-41df-bd7f-3e8fe90fb87b && \\\nTENANT_ID=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX && \\\nOIDC_BLOB_SERVICE_ENDPOINT=https://arohcpprowoidcj0074240.blob.core.windows.net/ && \\\nOIDC_ISSUER_BASE_ENDPOINT=https://arohcpprowoidcj0074240.z1.web.core.windows.net/ && \\\nDB_HOST=$(if [ \"false\" = \"true\" ]; then az postgres flexible-server show -g hcp-underlay-prow-j0074240 -n arohcp-prow-dbcs-j0074240 --query fullyQualifiedDomainName -o tsv; else echo \"ocm-cs-db\"; fi) && \\\nOVERRIDES=$(if [ \"false\" = \"true\" ]; then echo \"azuredb.values.yaml\"; else echo \"containerdb.values.yaml\"; fi) && \\\nZONE_RESOURCE_ID=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0074240/providers/Microsoft.Network/dnszones/j0074240.hcp.osadev.cloud && \\\nCX_SECRETS_KV_URL=\"https://ah-prow-cx-j0074240-1.vault.azure.net/\" && \\\nCX_MI_KV_URL=\"https://ah-prow-mi-j0074240-1.vault.azure.net/\" && \\\n./hack/helm.sh cluster-service helm-charts/cluster-service clusters-service \\\n -f helm-charts/cluster-service/${OVERRIDES} \\\n --set serviceAccountName=clusters-service \\\n --set environment=arohcpdev \\\n --set azureCsMiClientId=${AZURE_CS_MI_CLIENT_ID} \\\n --set oidcIssuerBlobServiceUrl=${OIDC_BLOB_SERVICE_ENDPOINT} \\\n --set oidcIssuerBaseUrl=${OIDC_ISSUER_BASE_ENDPOINT} \\\n --set tlsCertificatesIssuer=${TLS_CERTIFICATES_ISSUER} \\\n --set tlsCertificatesEnabled=true \\\n --set denyAssignments=disabled \\\n --set tenantId=${TENANT_ID} \\\n --set region=westus3 \\\n --set serviceKeyvaultName=aro-hcp-dev-svc-kv \\\n --set replicas=3 \\\n --set imageRegistry=arohcpsvcdev.azurecr.io \\\n --set imageRepository=app-sre/aro-hcp-clusters-service \\\n --set imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set global.imageDigest=sha256:877909c7b876e7cb8dd1d0cdb6485b92e9eb0bdd1494df44a308363b7a44b63b \\\n --set azureFirstPartyApplicationClientId=b3cb2fab-15cb-4583-ad06-f91da9bfe2d1 \\\n --set batchProcessesDryRun=true \\\n --set batchProcesses=\"\" \\\n --set fpaCertName=firstPartyCert2 \\\n --set ocpAcrResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev \\\n --set ocpAcrUrl=arohcpocpdev.azurecr.io \\\n --set shard.consumerName=\"hcp-underlay-j0074240-mgmt-1\" \\\n --set shard.zoneResourceId=\"${ZONE_RESOURCE_ID}\" \\\n --set shard.cxSecretsKeyVaultUrl=\"${CX_SECRETS_KV_URL}\" \\\n --set shard.cxMiKeyVaultUrl=\"${CX_MI_KV_URL}\" \\\n --set shard.cxSecretsKeyVaultMiClientId=\"c35d9cdb-c10a-4fd9-be0c-9292c8fe3d77\" \\\n --set shard.maestroRestUrl=\"http://maestro.maestro.svc.cluster.local:8000\" \\\n --set shard.maestroGrpUrl=\"maestro-grpc.maestro.svc.cluster.local:8090\" \\\n --set shard.Id=$(python3 -c \"import uuid; print(uuid.uuid5(uuid.UUID('916f9976-e1c0-4afd-b84c-5d5c94fbeaed'), 'prow-j0074240-mgmt-1'))\") \\\n --set shard.aksManagementClusterResourceId=/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j0074240-mgmt-1/providers/Microsoft.ContainerService/managedClusters/prow-j0074240-mgmt-1 \\\n --set databaseHost=${DB_HOST} \\\n --set azureMiMockServicePrincipalPrincipalId=d6b62dfa-87f5-49b3-bbcb-4a687c4faa96 \\\n --set azureMiMockServicePrincipalClientId=e8723db7-9b9e-46a4-9f7d-64d75c3534f0 \\\n --set azureMiMockServicePrincipalCertName=msiMockCert2 \\\n --set azureArmHelperIdentityCertName=armHelperCert2 \\\n --set azureArmHelperIdentityClientId=3331e670-0804-48e8-a086-6241671ddc93 \\\n --set azureArmHelperMockFpaPrincipalId=47f69502-0065-4d9a-b19b-d403e183d2f4 \\\n --set azureCloudEnvironmentName=AzurePublicCloud \\\n --set pullBinding.workloadIdentityClientId=\"${IMAGE_PULLER_MI_CLIENT_ID}\" \\\n --set pullBinding.workloadIdentityTenantId=\"${IMAGE_PULLER_MI_TENANT_ID}\" \\\n --set pullBinding.registry=arohcpsvcdev.azurecr.io \\\n --set pullBinding.scope=repository:app-sre/aro-hcp-clusters-service:pull \\\n --set managedIdentitiesDataPlaneAudienceResource=https://dummy.org \\\n --set tracing.address=http://ingest.observability:4318 \\\n --set csDeploymentStrategy.rollingUpdate.maxSurge=1 \\\n --set csDeploymentStrategy.rollingUpdate.maxUnavailable=0 \\\n --set provisionShardClusterLimit=100 \\\n --set ocpVersions.defaultVersion.version=4.20.8 \\\n --set ocpVersions.channelGroups.stable.minVersion=4.19.0 \\\n --set ocpVersions.channelGroups.stable.maxVersion= \\\n --set ocpVersions.channelGroups.candidate.minVersion=4.19.7 \\\n --set ocpVersions.channelGroups.candidate.maxVersion= \\\n --set ocpVersions.channelGroups.nightly.minVersion=4.19.0-0.nightly-20200101 \\\n --set ocpVersions.channelGroups.nightly.maxVersion= \\\n --set azureOperatorsMI.roleSetName=dev \\\n --set deployment.zoneCount=4\nnamespace/clusters-service created\nnamespace/clusters-service labeled\nRun Helm upgrade with release name cluster-service in namespace clusters-service\nRelease \"cluster-service\" does not exist. Installing it now.\nError: Kubernetes cluster unreachable: the server has asked for the client to provide credentials\nmake[2]: * [Makefile:32: deploy] Error 1\nmake[2]: Leaving directory '/opt/app-root/src/github.com/Azure/ARO-HCP/cluster-service'\n exit status 2"
16:12:33	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1c91345	n/a	provision	failed	⚠#4133 (open)	1	provision (1) Error: failed to apply tag webhook MutatingWebhookConfiguration to cluster: failed to apply tag manifests to cluster: patch: Unauthorized 16:12:33 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade · suite step graph Show raw failure time=2026-02-23T17:01:30.400Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: Shell\n Command: ./istio.sh\n" time=2026-02-23T17:01:38.385Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="error running Shell Step, failed to execute shell command: + set -euo pipefail\n+ ISTIO_NAMESPACE=aks-istio-system\n+ echo '******** Check istio is up and running **********'\n****** Check istio is up and running **********\n++ kubectl get pods -n aks-istio-system -l istio.io/rev=asm-1-26 --field-selector=status.phase=Running --no-headers\n++ wc -l\n+ ISTIO_PODS_COUNT=2\n+ [[ 2 -lt 2 ]]\n+ echo '****** Download istioctl **********'\n****** Download istioctl **********\n++ uname\n+ OS=Linux\n+ '[' Linux = Darwin ']'\n+ OSEXT=linux\n++ uname -m\n+ LOCAL_ARCH=x86_64\n+ case \"${LOCAL_ARCH}\" in\n+ ISTIO_ARCH=amd64\n+ ISTIO_URL=https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz\n+ SHA256_URL=https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz.sha256\n+ curl -sL https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz -o istio-1.28.3-linux-amd64.tar.gz\n+ curl -sL https://github.com/istio/istio/releases/download/1.28.3/istio-1.28.3-linux-amd64.tar.gz.sha256 -o istio-1.28.3-linux-amd64.tar.gz.sha256\n+ sha256sum -c istio-1.28.3-linux-amd64.tar.gz.sha256\nistio-1.28.3-linux-amd64.tar.gz: OK\n+ sha256sum -c istio-1.28.3-linux-amd64.tar.gz.sha256\nistio-1.28.3-linux-amd64.tar.gz: OK\n+ echo 'Verification successful: The file is intact.'\nVerification successful: The file is intact.\n+ tar -xzf istio-1.28.3-linux-amd64.tar.gz\n+ cd istio-1.28.3\n+ export PATH=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/scripts/istio-1.28.3/bin:/usr/local/bin:/usr/local/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/usr/local/bin:/usr/bin:/bin/dnf_wrapper:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/go/bin\n+ PATH=/opt/app-root/src/github.com/Azure/ARO-HCP/dev-infrastructure/scripts/istio-1.28.3/bin:/usr/local/bin:/usr/local/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/usr/local/bin:/usr/bin:/bin/dnf_wrapper:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/go/bin\n+ echo ==========================================================================\n==========================================================================\n+ echo '****** ISTIO IngressGateway IP Address assignment **********'\n****** ISTIO IngressGateway IP Address assignment **********\n+ ISTIO_IG_ANNOTATIONS='\n service.beta.kubernetes.io/azure-load-balancer-resource-group=hcp-underlay-prow-j5646592\n service.beta.kubernetes.io/azure-pip-name=aro-hcp-istio-ingress\n'\n+ for annotation in $ISTIO_IG_ANNOTATIONS\n+ kubectl annotate --overwrite svc aks-istio-ingressgateway-external service.beta.kubernetes.io/azure-load-balancer-resource-group=hcp-underlay-prow-j5646592 -n aks-istio-ingress\nservice/aks-istio-ingressgateway-external annotated\n+ for annotation in $ISTIO_IG_ANNOTATIONS\n+ kubectl annotate --overwrite svc aks-istio-ingressgateway-external service.beta.kubernetes.io/azure-pip-name=aro-hcp-istio-ingress -n aks-istio-ingress\nservice/aks-istio-ingressgateway-external annotated\n+ echo '****** ISTIO Upgrade **********'\n****** ISTIO Upgrade **********\n+ [[ -z asm-1-26 ]]\n+ NEWVERSION=asm-1-26\n+ echo '****** Istio Upgrade Started with version asm-1-26 **********'\n****** Istio Upgrade Started with version asm-1-26 ************\n+ istioctl tag set prod-stable --revision asm-1-26 --istioNamespace aks-istio-system --overwrite\nError: failed to apply tag webhook MutatingWebhookConfiguration to cluster: failed to apply tag manifests to cluster: patch: Unauthorized\n exit status 1"
16:06:04	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 3097be9 · merge 0ac5e19	n/a	provision	failed	★#4181 (merged)	1	provision (1) Error: Put "<url>": unexpected EOF 16:06:04 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.Frontend/global/mirror-image · suite step graph Show raw failure time=2026-02-23T16:57:03.568Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> frontend.image.registry <nil>}:{<nil> frontend.image.repository <nil>}@{<nil> frontend.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-23T16:57:26.823Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Frontend resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build06.ci.openshift.org\nSaved credentials for registry.build06.ci.openshift.org into /tmp/tmp.T2Z0933lgY/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build06.ci.openshift.org/ci-op-llzj9rp5/pipeline@sha256:9dcc9d7c2d8ff931fbaf2e6fcf42570c30f4e8a5864660849c177a57d8618c77 to arohcpsvcdev.azurecr.io/ci-op-llzj9rp5/pipeline:9dcc9d7c2d8ff931fbaf2e6fcf42570c30f4e8a5864660849c177a57d8618c77.\nThe image will still be available under it's original digest sha256:9dcc9d7c2d8ff931fbaf2e6fcf42570c30f4e8a5864660849c177a57d8618c77 in the target registry.\nError: Put \"https://arohcpsvcdev.azurecr.io/v2/ci-op-llzj9rp5/pipeline/blobs/uploads/559c7df2-c911-4f1d-964a-97467bb88fc7?_nouploadcache=false&_state=MrBGR3pxLqWIvVZ3aumY8KgsDo_dbQXGErN4moSdygJ7Ik5hbWUiOiJjaS1vcC1sbHpqOXJwNS9waXBlbGluZSIsIlVVSUQiOiI1NTljN2RmMi1jOTExLTRmMWQtOTY0YS05NzQ2N2JiODhmYzciLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjYtMDItMjNUMTY6NTc6MTMuMDE3NjIzOTIyWiJ9&digest=sha256%3A2711053d8bf70bfef173296492110e726b52a5d2282167461f5805e92bf002f5\": unexpected EOF\n exit status 1"
15:57:28	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 8c39c00 · merge 3ca804d	n/a	provision	failed	★#3679 (merged)	1	provision (1) failed to apply namespace monitoring: Unauthorized 15:57:28 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/kube-events · suite step graph Show raw failure time=2026-02-23T16:45:58.786Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events description="Step kube-events\n Kind: Helm\n" time=2026-02-23T16:45:59.343Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events values="map[enabled:true image:map[digest:sha256:6cf65c67321d5fc18e878715049a6ac8a8aea1ac2c63b096fbe18fade52f4be5 registry:kubernetesshared.azurecr.io repository:shared/kube-events]]" time=2026-02-23T16:45:59.343Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events time=2026-02-23T16:45:59.928Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace monitoring: Unauthorized"
15:45:49	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head a3e96f1 · merge e43ebba	n/a	unknown	failed	#4180 (merged)	1	non-artifact-backed failure (no junit artifacts)
15:44:38	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 2fb4e26 · merge 2fb4e26	n/a	e2e	failed	★#4067 (merged)	3	e2e (3) cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) 15:44:38 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc000608370>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc000608370>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift 15:44:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error: <errors.joinError \| 0xc00075b308>: failed to cleanup resource group: failed listing hcp clusters in resourcegroup="pullsecret-test-9vcbcb": GET https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-9vcbcb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <fmt.wrapError \| 0xc000e80ce0>{ msg: "failed to cleanup resource group: failed listing hcp clusters in resourcegroup=\"pullsecret-test-9vcbcb\": GET https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-9vcbcb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e80cc0>{ msg: "failed listing hcp clusters in resourcegroup=\"pullsecret-test-9vcbcb\": GET https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-9vcbcb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000b7f530>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "d39ef905-0aed-4125-b65d-37110da01bf2", ], "Date": [ "Mon, 23 Feb 2026 17:10:35 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["3138"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000e80c00>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error: <errors.joinError \| 0xc00075b308>: failed to cleanup resource group: failed listing hcp clusters in resourcegroup="pullsecret-test-9vcbcb": GET https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-9vcbcb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters -------------------------------------------------------------------------------- RESPONSE 500: 500 Internal Server Error ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "error": { "code": "InternalServerError", "message": "Internal server error." } } -------------------------------------------------------------------------------- { errs: [ <fmt.wrapError \| 0xc000e80ce0>{ msg: "failed to cleanup resource group: failed listing hcp clusters in resourcegroup=\"pullsecret-test-9vcbcb\": GET https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-9vcbcb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e80cc0>{ msg: "failed listing hcp clusters in resourcegroup=\"pullsecret-test-9vcbcb\": GET https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-9vcbcb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Internal server error.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000b7f530>{ ErrorCode: "InternalServerError", StatusCode: 500, RawResponse: { Status: "500 Internal Server Error", StatusCode: 500, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Error-Code": [ "InternalServerError", ], "X-Ms-Request-Id": [ "d39ef905-0aed-4125-b65d-37110da01bf2", ], "Date": [ "Mon, 23 Feb 2026 17:10:35 GMT", ], "Content-Length": ["107"], "X-Envoy-Upstream-Service-Time": ["3138"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000e80c00>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 107, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { Method: "GET", URL: { Scheme: "https", Opaque: "", User... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift 15:44:38 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapError \| 0xc000517a00>: failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-cczltc": PUT https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-cczltc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: LockContention -------------------------------------------------------------------------------- { "error": { "code": "LockContention", "message": "Failed to acquire lock: timed out", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-tags-cluster: failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-cczltc\": PUT https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-cczltc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005179e0>{ msg: "failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-cczltc\": PUT https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-cczltc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000687a70>{ ErrorCode: "LockContention", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "Retry-After": ["10"], "X-Ms-Error-Code": ["LockContention"], "X-Ms-Request-Id": [ "cc51ddc9-494c-46d3-903b-e4181bb17e80", ], "Date": [ "Mon, 23 Feb 2026 17:11:19 GMT", ], "Content-Length": ["186"], "X-Envoy-Upstream-Service-Time": ["10052"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000517920>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 186, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapError \| 0xc000517a00>: failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-cczltc": PUT https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-cczltc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 503: 503 Service Unavailable ERROR CODE: LockContention -------------------------------------------------------------------------------- { "error": { "code": "LockContention", "message": "Failed to acquire lock: timed out", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-tags-cluster: failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-cczltc\": PUT https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-cczltc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0005179e0>{ msg: "failed starting cluster creation \"patch-tags-cluster\" in resourcegroup=\"patch-tags-cczltc\": PUT https://rp.j4882048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-cczltc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"LockContention\",\n \"message\": \"Failed to acquire lock: timed out\",\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000687a70>{ ErrorCode: "LockContention", StatusCode: 503, RawResponse: { Status: "503 Service Unavailable", StatusCode: 503, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "Retry-After": ["10"], "X-Ms-Error-Code": ["LockContention"], "X-Ms-Request-Id": [ "cc51ddc9-494c-46d3-903b-e4181bb17e80", ], "Date": [ "Mon, 23 Feb 2026 17:11:19 GMT", ], "Content-Length": ["186"], "X-Envoy-Upstream-Service-Time": ["10052"], "Server": ["istio-envoy"], }, Body: <exported.nopClosingBytesReader \| 0xc000517920>{ s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 34, 10, 32, 32, 32, 32, 125, 10, 125], i: 0, }, ContentLength: 186, TransferEncoding: nil, Close: false, Uncompressed: false, Trailer: nil, Request: { ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
15:30:44	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 1a8a9e6 · merge 1a8a9e6	n/a	provision	failed	★#4111 (merged)	1	provision (1) failed to install CRD crds/crd-podmonitors.yaml: Unauthorized 15:30:44 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite step graph Show raw failure time=2026-02-23T16:30:36.376Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-02-23T16:30:36.992Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpdev kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:c3621782e405e59b40e1b64e8b8722855f5c348a9b6fe3ac58c54ad1a34afbb9]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen tag:v1.5.2]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:f97255b7629052f03e21d16171923e7095a9608526196af59e2989a1eb01af9c] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:f9a0f85d9a348f7f83e4b1fcd602eb12fcee0c83b0f7e4eb77d205f6af4311c9]] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[prometheusSpec:map[replicas:2 shards:1] serviceAccount:map[managedIdentity:835dceac-e813-4165-aa7f-68983589d51e]] prometheusSpec:map[externalLabels:map[cluster:prow-j2320384-svc] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:375d727bf8137d574b5aad4f94881fade0a57556ddf52a3f98322e5d561286af] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-prow-j2320384-svc-93q5.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-da583ed801424d28b4d3baf92aef9e9b/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 resources:map[limits:map[cpu:NONE memory:NONE] requests:map[cpu:NONE memory:NONE]] retention:1d retentionSize:45GiB version: zoneCount:4]]" time=2026-02-23T16:30:36.992Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-02-23T16:30:37.691Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus time=2026-02-23T16:30:37.691Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false time=2026-02-23T16:30:37.691Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-02-23T16:30:38.039Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-02-23T16:30:39.964Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed to install CRD crds/crd-podmonitors.yaml: Unauthorized" time=2026-02-23T16:30:39.964Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-02-23T16:30:39.964Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-02-23T16:30:40.096Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed to get status for release arohcp-monitor: release: not found" time=2026-02-23T16:30:40.096Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-02-23T16:30:40.096Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to install CRD crds/crd-podmonitors.yaml: Unauthorized"
15:29:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 6b5fd66 · merge 6b5fd66	n/a	provision	failed	★#4015 (merged)	1	provision (1) Unauthorized Unauthorized Unauthorized 15:29:07 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Agent/management/deploy · suite step graph Show raw failure time=2026-02-23T16:20:32.647Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy description="Step deploy\n Kind: Helm\n" time=2026-02-23T16:20:33.268Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy values="map[azure:map[clientId:ae3a00ed-a015-4cab-831d-b4f98aa4f2f6 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-prow-maestro-j9727104.westus3-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j9727104-mgmt-1 credsKeyVault:map[name:ah-prow-mg-j9727104-1 secret:hcp-underlay-j9727104-mgmt-1] deployment:map[zoneCount:4] glog_v:2 image:map[digest:sha256:8d336b3025d6557ed1cbd98f9979593e3693ab2f9a21bc7bfc183c6f2b5bc382 registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:3 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:2dccfca276d1e7611442dc997ac15055f76c4caff92b668b40907cdd68fffa14 registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]" time=2026-02-23T16:20:33.268Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-02-23T16:20:33.527Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy namespace=maestro time=2026-02-23T16:20:33.527Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy dryRun=false time=2026-02-23T16:20:33.528Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-02-23T16:20:33.843Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-02-23T16:20:41.976Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy err="Unauthorized\nUnauthorized\nUnauthorized" time=2026-02-23T16:20:41.976Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-02-23T16:20:41.976Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-02-23T16:20:45.229Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy release=maestro-agent namespace=maestro status=failed description="Release \"maestro-agent\" failed: Unauthorized\nUnauthorized\nUnauthorized" values="map[azure:map[clientId:ae3a00ed-a015-4cab-831d-b4f98aa4f2f6 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] broker:map[host:arohcp-prow-maestro-j9727104.westus3-1.ts.eventgrid.azure.net port:8883] consumerName:hcp-underlay-j9727104-mgmt-1 credsKeyVault:map[name:ah-prow-mg-j9727104-1 secret:hcp-underlay-j9727104-mgmt-1] deployment:map[zoneCount:4] glog_v:2 image:map[digest:sha256:8d336b3025d6557ed1cbd98f9979593e3693ab2f9a21bc7bfc183c6f2b5bc382 registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] installAppliedManifestWorkCRD:false replicas:3 serviceAccountName:maestro sideCar:map[image:map[digest:sha256:2dccfca276d1e7611442dc997ac15055f76c4caff92b668b40907cdd68fffa14 registry:mcr.microsoft.com repository:azurelinux/base/nginx]]]" time=2026-02-23T16:20:45.230Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy url="https://hcp-dev-us.westus3.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6xTwW7TQBC9+ytGPiVSXAUHejDqhYoTEkLAqYjDevfJbGPPWjOTNJHy8chJoS5NQqX2Yllv37w3uzMva2Ek0LQSD6UrCs6cubrFJNW38PYpcqjUJHIzowP02XX4C2EN3iPaO/8Hnv7IiIjyb/ACy2eUdw5qkh79PuHAJHotnPdQLSwtwUcKrtuVGuRrajE6LVwDtmr/LR5hA+lU7b1jL2mzvfc9Qf8QOURuXuJ4UgIb+JXFxIULXeRXEhsXnnmsChsD6969jwpZQ552sFzVKHSrhu4Zmkem9v83fFEj56VPrt0XSesYINetO0z+yJ5mP99ng6swDPpx2HfNdnT3CwKy2EHNdT3VsDuAaRKcYYAneTkvL4t5WZSL728uq3JeLRY3+ZQuLugM5+27m3w6zXZ0myLTMnK4isyQUUoT00M4x6n8N47ZjnpJw+lDozPqUziQnycyozY1M+qg6ppBMkmAUL0dXT5A/e8AAAD//6sx2i5LBAAA" time=2026-02-23T16:20:45.230Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy time=2026-02-23T16:20:45.231Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=management step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Unauthorized\nUnauthorized\nUnauthorized"
15:00:29	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 33c7d53 · merge 31af859	n/a	provision	failed	⚠#4041 (merged)	1	provision (1) failed to apply namespace arobit: an error on the server ("Internal Server Error: \\"/api/v1/namespaces/arobit?fieldManager=templatize&force=false\\": Post \\"<url>": proxy error from localhost:9443 while dialing 15:00:29 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/arobit · suite step graph Show raw failure time=2026-02-23T15:46:31.506Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit description="Step arobit\n Kind: Helm\n" time=2026-02-23T15:46:32.210Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:prow-j5736192-mgmt-1 clusterType:mgmt dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eca5fdd2a205c5e2a78155ca9d16b453334c5aefd6cafc48465c86ed9c016b64 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit]] geoShortId:us kusto:map[enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us.westus3.kusto.windows.net serviceLogsDatabase:ServiceLogs] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan: clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-prow-j5736192-mgmt-1 rpAccountName:placeholder rpNamespace: rpSan: rpSecretName:rplogs] image:map[digest:sha256:18d5476700adcd40e9ee2bc1cb62a1150ac1b5e696d7bad6adeea4da88c615db pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:afd8f34b-37ea-40eb-a7a9-8f9dbc2afdff region:westus3 secretProvider:map[keyVault:ah-prow-mg-j5736192-1] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]" time=2026-02-23T15:46:32.211Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit time=2026-02-23T15:46:32.309Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace arobit: an error on the server (\"Internal Server Error: \\\"/api/v1/namespaces/arobit?fieldManager=templatize&force=false\\\": Post \\\"https://guard.699c73bbb0620a0001af8c96.svc.cluster.local:443/subjectaccessreviews?timeout=30s\\\": proxy error from localhost:9443 while dialing guard.699c73bbb0620a0001af8c96.svc.cluster.local:443, code 503: 503 Service Unavailable\") has prevented the request from succeeding (patch namespaces arobit)"
14:49:43	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 2786b84 · merge e6b55da	n/a	unknown	failed	#4166 (merged)	1	non-artifact-backed failure (no junit artifacts)
14:12:58	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 0cf0aa6 · merge 463a6d0	n/a	provision	failed	★#4055 (merged)	1	provision (1) failed to apply namespace monitoring: Unauthorized 14:12:58 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/kube-events · suite step graph Show raw failure time=2026-02-23T15:01:24.166Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events description="Step kube-events\n Kind: Helm\n" time=2026-02-23T15:01:24.910Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events values="map[enabled:true image:map[digest:sha256:6cf65c67321d5fc18e878715049a6ac8a8aea1ac2c63b096fbe18fade52f4be5 registry:kubernetesshared.azurecr.io repository:shared/kube-events]]" time=2026-02-23T15:01:24.910Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events time=2026-02-23T15:01:25.559Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=kube-events err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace monitoring: Unauthorized"
13:57:28	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1c91345	n/a	provision	failed	⚠#4133 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code OperationNotAllowed; detail message Preflight validation check for resource(s) for container service prow-j3474816-svc in resource group <resource-group> failed. Message: Operation is not allowed because there's an in progress upgrade managed cluster operation (operation ID: <uuid>) on the managed cluster started on UTC <timestamp>. 13:57:28 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite step graph Show raw failure time=2026-02-23T14:39:46.286Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-02-23T14:39:48.717Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-02-23T14:39:50.484Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=e92ad84584aa354238bd4c036e5c9344ad12d32f609be32548c270b3602f9959 time=2026-02-23T15:01:55.999Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j3474816-svc/providers/Microsoft.Resources/deployments/e92ad84584aa354238bd4c036e5c9344ad12d32f609be32548c270b3602f9959/operationStatuses/08584297492959011143\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3474816-svc/providers/Microsoft.Resources/deployments/maestro-server\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/maestro-eg-crt-4kqobph52xogo\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/maestro-eg-crt-4kqobph52xogo\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/maestro-server-j3474816-cert\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deployments/maestro-server-j3474816-cert\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.Resources/deploymentScripts/newCertwithRotationKV-maestro-server-j3474816\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentScriptACIProvisioningTimeout\\\",\\r\\n \\\"message\\\": \\\"The supporting resource: 'Azure Container Instance' did not provision during the expected time. Correlation Id: c9214140-3783-47fc-ad4b-bef618bafaa3. Please try again later, if the issue persists contact technical support for further investigation.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" time=2026-02-23T15:03:56.015Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-02-23T15:03:58.106Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-02-23T15:03:59.647Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=4ff6f488d089e284097b5d30588d17bb42279ee9ca15dfea9989c3c6048e99e6 time=2026-02-23T15:08:00.873Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j3474816-svc/providers/Microsoft.Resources/deployments/4ff6f488d089e284097b5d30588d17bb42279ee9ca15dfea9989c3c6048e99e6/operationStatuses/08584297478466819714\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"InvalidTemplateDeployment\\\",\\r\\n \\\"message\\\": \\\"The template deployment 'cluster-fbcje44hvaogk' is not valid according to the validation procedure. The tracking id is '53602e69-5e0b-4c72-bde8-ec2c30d603b0'. See inner errors for details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"OperationNotAllowed\\\",\\r\\n \\\"message\\\": \\\"Preflight validation check for resource(s) for container service prow-j3474816-svc in resource group hcp-underlay-prow-j3474816-svc failed. Message: Operation is not allowed because there's an in progress upgrade managed cluster operation (operation ID: 28392fb7-516e-478e-9202-f09223b923a5) on the managed cluster started on UTC 2026-02-23T15:00:28Z. Please wait for it to finish before starting a new operation. You can also use 'az aks operation-abort ...' to abort the ongoing operation.. Details: \\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
13:55:46	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head e247c0c · merge 2f8a0eb	n/a	n/a	passed	#4152 (merged)	0	n/a
13:21:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 2fb4e26 · merge 2fb4e26	n/a	e2e	failed	★#4067 (merged)	1	e2e (1) cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) 13:21:14 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc00059dc50>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc00059dc50>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred
13:18:26	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 2fb4e26 · merge 2fb4e26	n/a	e2e	failed	★#4067 (merged)	1	e2e (1) cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) 13:18:26 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc00050d0b0>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc00050d0b0>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred
13:01:29	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4bea39e	n/a	e2e	failed	⚠#3094 (closed)	1	e2e (1) subscription state not found 13:01:29 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:314]: Timed out after 300.011s. NFD operator should be installed successfully Expected success, but got an error: <errors.errorString \| 0xc000373400>: subscription state not found { s: "subscription state not found", } fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:314]: Timed out after 300.011s. NFD operator should be installed successfully Expected success, but got an error: <errors.errorString \| 0xc000373400>: subscription state not found { s: "subscription state not found", }
12:48:05	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 1b1dffe · merge 6b5fd66	n/a	n/a	passed	#4015 (merged)	0	n/a
12:45:22	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 1b1dffe · merge 6b5fd66	n/a	n/a	passed	#4015 (merged)	0	n/a
12:29:48	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 1ef755c · merge 3ca804d	n/a	provision	failed	⚠#3679 (merged)	1	provision (1) Error response from registry: generic storage error: unknown_storage_failure 12:29:48 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SessionGate/global/mirror-image · suite step graph Show raw failure time=2026-02-23T13:11:14.734Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> sessiongate.image.registry <nil>}:{<nil> sessiongate.image.repository <nil>}@{<nil> sessiongate.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-23T13:11:53.626Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build06.ci.openshift.org\nSaved credentials for registry.build06.ci.openshift.org into /tmp/tmp.mHI7jJKp2K/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build06.ci.openshift.org/ci-op-5ldcv2vq/pipeline@sha256:b020aeb639887c826d9c6fc142926295936d1c904ab58bc5eafc5e60079dbb7d to arohcpsvcdev.azurecr.io/ci-op-5ldcv2vq/pipeline:b020aeb639887c826d9c6fc142926295936d1c904ab58bc5eafc5e60079dbb7d.\nThe image will still be available under it's original digest sha256:b020aeb639887c826d9c6fc142926295936d1c904ab58bc5eafc5e60079dbb7d in the target registry.\nError response from registry: generic storage error: unknown_storage_failure\n exit status 1"
11:21:12	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 1b1dffe · merge 6b5fd66	n/a	n/a	passed	#4015 (merged)	0	n/a
11:15:35	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 0cf0aa6 · merge 463a6d0	n/a	e2e	failed	⚠#4055 (merged)	2	e2e (2) missing expected log sources: [ocm-arohcpdev-/kube-apiserver] 11:15:35 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc000977770>: missing expected log sources: [ocm-arohcpdev-/kube-apiserver] { s: "missing expected log sources: [ocm-arohcpdev-/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.001s. Expected success, but got an error: <errors.errorString \| 0xc000977770>: missing expected log sources: [ocm-arohcpdev-/kube-apiserver] { s: "missing expected log sources: [ocm-arohcpdev-/kube-apiserver]", } cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) 11:15:35 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc000383400>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc000383400>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred
10:51:10	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 7dd671c · merge 2714126	n/a	unknown	failed	#4056 (merged)	1	non-artifact-backed failure (no junit artifacts)
10:24:54	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 926b275 · merge 926b275	n/a	n/a	passed	#4170 (merged)	0	n/a
10:22:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 02412ae · merge 02412ae	n/a	n/a	passed	#4155 (merged)	0	n/a
10:06:39	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 1a8a9e6 · merge 1a8a9e6	n/a	n/a	passed	#4111 (merged)	0	n/a
09:25:50	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 1a8a9e6 · merge 1a8a9e6	n/a	provision	failed	★#4111 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid 09:25:50 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph Show raw failure time=2026-02-23T09:52:00.476Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-02-23T09:52:02.591Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra time=2026-02-23T09:52:03.585Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=d615e243da6b4edf4108a39931b232025a7d8e2f5406d864f950aaaa8f85389e time=2026-02-23T09:53:34.450Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j3043584/providers/Microsoft.Resources/deployments/d615e243da6b4edf4108a39931b232025a7d8e2f5406d864f950aaaa8f85389e/operationStatuses/08584297665625794640\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3043584/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3043584/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j3043584\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report b38c185f-f016-41e7-bc83-2f044577152a:2/23/2026 9:53:06 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
09:16:38	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head da15e65 · merge 97dcd34	n/a	provision	failed	⚠#4128 (merged)	1	provision (1) unable to build kubernetes objects from release manifest: resource mapping not found for name: "aro-hcp-exporter" namespace: "aro-hcp-exporter" from "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" ensure CRDs are installed first 09:16:38 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/aro-hcp-exporter · suite step graph Show raw failure time=2026-02-23T10:00:06.581Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter description="Step aro-hcp-exporter\n Kind: Helm\n" time=2026-02-23T10:00:07.159Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter values="map[cacheTTL:60m collectionInterval:2m deployment:map[name:aro-hcp-exporter] enabledCollectors:[service-tag-usage] image:map[digest:sha256:b138301009f792415bbd9283194d5142126db46886f3dd7ecd6f6ba855e53c18 registry:arohcpsvcdev.azurecr.io repository:arohcpmetrics] listenAddress::8080 service:map[name:aro-hcp-exporter] serviceAccount:map[create:true msiClientId:18773e74-1b2a-4a69-b4bf-f17ad02f9791 name:aro-hcp-exporter] subscriptionNames:[XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]" time=2026-02-23T10:00:07.159Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter time=2026-02-23T10:00:07.449Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter namespace=aro-hcp-exporter time=2026-02-23T10:00:07.449Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter dryRun=false time=2026-02-23T10:00:07.449Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter time=2026-02-23T10:00:07.828Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter time=2026-02-23T10:00:10.238Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter err="unable to build kubernetes objects from release manifest: resource mapping not found for name: \"aro-hcp-exporter\" namespace: \"aro-hcp-exporter\" from \"\": no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"\nensure CRDs are installed first" time=2026-02-23T10:00:10.238Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter time=2026-02-23T10:00:10.238Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter time=2026-02-23T10:00:10.354Z level=ERROR msg="Failed to capture diagnostics for Helm release" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter err="failed to get status for release aro-hcp-exporter: release: not found" time=2026-02-23T10:00:10.354Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter time=2026-02-23T10:00:10.354Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=aro-hcp-exporter err="error running Helm release deployment Step, failed to deploy helm release: unable to build kubernetes objects from release manifest: resource mapping not found for name: \"aro-hcp-exporter\" namespace: \"aro-hcp-exporter\" from \"\": no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"\nensure CRDs are installed first"
09:06:02	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head a0c4bfd · merge 2714126	n/a	unknown	failed	#4056 (merged)	1	non-artifact-backed failure (no junit artifacts)
08:54:21	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 0cf0aa6 · merge 463a6d0	n/a	e2e	failed	⚠#4055 (merged)	2	e2e (2) VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then 08:54:21 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:159]: Unexpected error: <errors.joinError \| 0xc000911158>: VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then { errs: [ <fmt.wrapError \| 0xc0004109c0>{ msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", err: <errors.errorString \| 0xc0003c9a30>{ s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", }, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:159]: Unexpected error: <errors.joinError \| 0xc000911158>: VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then { errs: [ <fmt.wrapError \| 0xc0004109c0>{ msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", err: <errors.errorString \| 0xc0003c9a30>{ s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", }, }, ], } occurred missing expected log sources: [ocm-arohcpdev-/kube-apiserver] 08:54:21 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.010s. Expected success, but got an error: <errors.errorString \| 0xc0007798d0>: missing expected log sources: [ocm-arohcpdev-/kube-apiserver] { s: "missing expected log sources: [ocm-arohcpdev-/kube-apiserver]", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.010s. Expected success, but got an error: <errors.errorString \| 0xc0007798d0>: missing expected log sources: [ocm-arohcpdev-/kube-apiserver] { s: "missing expected log sources: [ocm-arohcpdev-/kube-apiserver]", }
08:47:48	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 2fb4e26 · merge 2fb4e26	n/a	n/a	passed	#4067 (merged)	0	n/a
08:47:38	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 1b1dffe · merge 6b5fd66	n/a	n/a	passed	#4015 (merged)	0	n/a
07:56:52	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 02412ae · merge 02412ae	n/a	n/a	passed	#4155 (merged)	0	n/a
07:29:29	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head b1e9af9 · merge b1e9af9	n/a	n/a	passed	#4153 (merged)	0	n/a
07:08:37	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 926b275 · merge 926b275	n/a	e2e	failed	★#4170 (merged)	1	e2e (1) All ClusterOperators should report Available=True, but these are not available: [monitoring (Available=Unknown)] 07:08:37 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.004s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [monitoring (Available=Unknown)] Expected <[]string \| len:1, cap:1>: [ "monitoring (Available=Unknown)", ] to be empty fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:390]: Timed out after 600.004s. The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:389 with: All ClusterOperators should report Available=True, but these are not available: [monitoring (Available=Unknown)] Expected <[]string \| len:1, cap:1>: [ "monitoring (Available=Unknown)", ] to be empty
05:46:38	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0a94a4c	n/a	n/a	passed	#4178 (closed)	0	n/a
04:55:24	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head bd23425 · merge bd23425	n/a	n/a	passed	#4148 (merged)	0	n/a
04:35:44	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e1e8f06 · merge e1e8f06	n/a	e2e	failed	★#4144 (merged)	1	e2e (1) cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) 04:35:44 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc000097ad0>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc000097ad0>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred
04:05:28	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head e1e8f06 · merge e1e8f06	n/a	e2e	failed	★#4144 (merged)	1	e2e (1) cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) 04:05:28 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc0004e6f50>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc0004e6f50>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred
04:02:39	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:59:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:57:10	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:54:27	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:51:43	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:48:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:46:13	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:42:54	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:40:09	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:37:24	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:34:40	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:31:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:29:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:26:27	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:23:41	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:20:54	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:18:10	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:15:20	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:12:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:09:54	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:07:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:04:21	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
03:01:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:58:53	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:56:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:53:20	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:50:36	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:47:54	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:45:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:42:03	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:39:13	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:37:33	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 5d6ec85 · merge b1e9af9	n/a	n/a	passed	#4153 (merged)	0	n/a
02:36:31	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:33:50	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:30:59	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:28:15	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:25:33	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:22:49	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:20:02	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:17:12	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:15:28	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 6b7b9b8	n/a	provision	failed	★#4133 (open)	1	provision (1) Error: Put "<url>": unexpected EOF 02:15:28 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.AdminAPI/global/mirror-image · suite step graph Show raw failure time=2026-02-23T03:06:00.325Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=global step=mirror-image description="Step mirror-image\n Kind: ImageMirror\n From {<nil> adminApi.image.registry <nil>}:{<nil> adminApi.image.repository <nil>}@{<nil> adminApi.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n" time=2026-02-23T03:12:45.695Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.AdminAPI resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org registry.build06.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build06.ci.openshift.org\nSaved credentials for registry.build06.ci.openshift.org into /tmp/tmp.EBeKxKnzpB/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image registry.build06.ci.openshift.org/ci-op-09tlr81s/pipeline@sha256:ffec63e977a8259c668073c3abbd8da45f08db699d78368d9ffb04305b70a44c to arohcpsvcdev.azurecr.io/ci-op-09tlr81s/pipeline:ffec63e977a8259c668073c3abbd8da45f08db699d78368d9ffb04305b70a44c.\nThe image will still be available under it's original digest sha256:ffec63e977a8259c668073c3abbd8da45f08db699d78368d9ffb04305b70a44c in the target registry.\nError: Put \"https://arohcpsvcdev.azurecr.io/v2/ci-op-09tlr81s/pipeline/blobs/uploads/6b522657-f711-4b8d-9f1c-51f4c2725e3d?_nouploadcache=false&_state=WDnv6548LfSd9TNNOikeSDDFWRdRDEmPHw3X28ZIzNx7Ik5hbWUiOiJjaS1vcC0wOXRscjgxcy9waXBlbGluZSIsIlVVSUQiOiI2YjUyMjY1Ny1mNzExLTRiOGQtOWYxYy01MWY0YzI3MjVlM2QiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjYtMDItMjNUMDM6MDY6MDMuNDA5MTcwMjk0WiJ9&digest=sha256%3Aaa93729c69e7598112bbe8a81426b7ce53875c722c34c04aed76fa226fa103ba\": unexpected EOF\n exit status 1"
02:14:29	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:11:42	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:09:01	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:06:14	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:03:28	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
02:00:41	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:57:56	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:55:10	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:52:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:49:40	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:46:53	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:43:42	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:40:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:38:07	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:35:25	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:32:38	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:29:53	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:27:08	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:24:25	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:21:39	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:18:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:16:12	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:13:23	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:10:42	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:07:55	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
01:05:09	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
00:41:34	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 1540346 · merge 1540346	n/a	n/a	passed	#4175 (merged)	0	n/a
00:39:54	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 4db6d63 · merge 1a8a9e6	n/a	unknown	failed	#4111 (merged)	1	non-artifact-backed failure (no junit artifacts)
00:20:32	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 33c7d53 · merge 31af859	n/a	e2e	failed	★#4041 (merged)	2	e2e (2) VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then 00:20:32 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:108]: Unexpected error: <errors.joinError \| 0xc00017ddb8>: VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then { errs: [ <fmt.wrapError \| 0xc00069b3a0>{ msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", err: <errors.errorString \| 0xc0005d58d0>{ s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", }, }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:108]: Unexpected error: <errors.joinError \| 0xc00017ddb8>: VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then { errs: [ <fmt.wrapError \| 0xc00069b3a0>{ msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", err: <errors.errorString \| 0xc0005d58d0>{ s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then", }, }, ], } occurred cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) 00:20:32 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc0005cd170>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:242]: Unexpected error: <errors.errorString \| 0xc0005cd170>: cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown) { s: "cluster operators not available: monitoring (Available=Unknown, Degraded=Unknown, Progressing=True, Upgradeable=Unknown)", } occurred
00:17:51	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel batch head 33c7d53 · merge 31af859	n/a	unknown	failed	#4041 (merged)	1	non-artifact-backed failure (no junit artifacts)
00:06:57	n/a	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a2896ee	n/a	provision	failed	★#3913 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid 00:06:57 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph Show raw failure time=2026-02-23T00:33:23.914Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n Kind: ARM\n Template: templates/region.bicep\n Parameters: configurations/region.tmpl.bicepparam" time=2026-02-23T00:33:26.337Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra time=2026-02-23T00:33:27.500Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=df333c44ccf2c4b6827592baee8b784e74ca66371b75131a52288da51c28b79c time=2026-02-23T00:34:58.228Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5879808/providers/Microsoft.Resources/deployments/df333c44ccf2c4b6827592baee8b784e74ca66371b75131a52288da51c28b79c/operationStatuses/08584298000786465513\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5879808/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5879808/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j5879808\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"Internal error\\\",\\r\\n \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 93a6abdd-3afc-444a-b7c8-61d62e9d3102:2/23/2026 12:34:30 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"