CIHealth
<
Single Day: Mar 18▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 39d25fe · merge 39d25fe
 n/a n/a passed #4511 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 732c175 · merge 732c175
 n/a e2e failed #4396 (merged) 1
e2e (1)
port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Error
23:36:54 UTC · failed at e2e · test Engineering should be able to retrieve expected metrics from the /metrics endpoint · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
    <*fmt.wrapError | 0xc000800ae0>: 
    port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error
    {
        msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
        err: <*errors.errorString | 0xc000a68cd0>{
            s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/exporter_metrics.go:74]: Failed to set up port-forward
Unexpected error:
    <*fmt.wrapError | 0xc000800ae0>: 
    port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error
    {
        msg: "port-forward failed: error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
        err: <*errors.errorString | 0xc000a68cd0>{
            s: "error upgrading connection: error dialing backend: proxy error from localhost:9443 while dialing 10.128.8.8:10250, code 500: 500 Internal Server Error",
        },
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 8a3e267 · merge 8a3e267
 n/a n/a passed #4493 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0928008 · merge 0928008
 n/a n/a passed #4444 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9cfdfc7
 n/a n/a passed #4484 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head f27ad30 · merge ec8c17a
 n/a e2e failed ⚠#4505 (merged) 1
e2e (1)
<string>: 4.21.5
21:52:43 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20 to 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_minor_upgrade.go:178]: Expected
    <string>: 4.20.16
to equal
    <string>: 4.21.5
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_minor_upgrade.go:178]: Expected
    <string>: 4.20.16
to equal
    <string>: 4.21.5
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head a5c42f1 · merge 98f9b5f
 n/a n/a passed #4517 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 39d25fe · merge 39d25fe
 n/a n/a passed #4511 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ca7bbc9 · merge ca7bbc9
 n/a n/a passed #4512 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 732c175 · merge 732c175
 n/a n/a passed #4396 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c84eed0 · merge 9feee09
 n/a n/a passed #4004 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 8550ece · merge 8550ece
 n/a n/a passed #4219 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 04f43f0 · merge 04f43f0
 n/a n/a passed #4482 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 3346aec · merge 3346aec
 n/a e2e failed ★#4519 (merged) 1
e2e (1)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
19:50:24 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s.
Expected success, but got an error:
    <*errors.errorString | 0xc000da0f80>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s.
Expected success, but got an error:
    <*errors.errorString | 0xc000da0f80>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head dacf876
 n/a unknown failed #4391 (open) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head dacf876
 n/a unknown failed #4391 (open) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head ea2b721 · merge ec8c17a
 n/a e2e failed ⚠#4505 (merged) 2
e2e (2)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
18:38:45 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0xc000a077d0>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0xc000a077d0>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
<string>: 4.21.5
18:38:45 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20 to 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_minor_upgrade.go:180]: Expected
    <string>: 4.20.16
to equal
    <string>: 4.21.5
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_minor_upgrade.go:180]: Expected
    <string>: 4.20.16
to equal
    <string>: 4.21.5
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head a5c42f1 · merge 98f9b5f
 n/a e2e failed ★#4517 (merged) 2
e2e (2)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
18:35:55 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s.
Expected success, but got an error:
    <*errors.errorString | 0xc000511410>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s.
Expected success, but got an error:
    <*errors.errorString | 0xc000511410>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then
18:35:55 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:107]: Unexpected error:
    <*errors.joinError | 0xc000c48e70>: 
    VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then
    {
        errs: [
            <*fmt.wrapError | 0xc0012fb220>{
                msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
                err: <*errors.errorString | 0xc000a0b210>{
                    s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:107]: Unexpected error:
    <*errors.joinError | 0xc000c48e70>: 
    VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then
    {
        errs: [
            <*fmt.wrapError | 0xc0012fb220>{
                msg: "VerifyAllAPIServicesAvailable failed: the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
                err: <*errors.errorString | 0xc000a0b210>{
                    s: "the APIService verifier is not allowed to re-try after Jan 15th, 2026 - HyperShift should only post ready status once aggregated APIs are ready then",
                },
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 6b929a5 · merge 4f10ef5
 n/a e2e failed ⚠#4133 (merged) 2
e2e (2)
ERROR CODE: InvalidRequestContent; detail message Replicas+Autoscaling.Max: The total number of compute nodes for a single cluster '700' exceeds the maximum allowed '500'. Reduce the total compute nodes requested to be within the maximum allowed.; provider Microsoft.RedHatOpenShift
18:32:41 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to 500 replicas · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:134]: Unexpected error:
    <*fmt.wrapError | 0xc000962c60>: 
    failed to create NodePool autoscale-az: failed starting nodepool creation "autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-9ntvp4": PUT https://rp.j6696320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9ntvp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Replicas+Autoscaling.Max: The total number of compute nodes for a single cluster '700' exceeds the maximum allowed '500'. Reduce the total compute nodes requested to be within the maximum allowed."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool autoscale-az: failed starting nodepool creation \"autoscale-az\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9ntvp4\": PUT https://rp.j6696320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9ntvp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Replicas+Autoscaling.Max: The total number of compute nodes for a single cluster '700' exceeds the maximum allowed '500'. Reduce the total compute nodes requested to be within the maximum allowed.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000962c40>{
            msg: "failed starting nodepool creation \"autoscale-az\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9ntvp4\": PUT https://rp.j6696320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9ntvp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Replicas+Autoscaling.Max: The total number of compute nodes for a single cluster '700' exceeds the maximum allowed '500'. Reduce the total compute nodes requested to be within the maximum allowed.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000b967b0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "0db94d04-dd8e-47c4-a152-06aa92aab279",
                        ],
                        "Date": [
                            "Wed, 18 Mar 2026 19:48:49 GMT",
                        ],
                        "Content-Length": ["283"],
                        "X-Envoy-Upstream-Service-Time": ["1179"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000962b80>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 112, 108, 105, 99, 97, 115, 43, 65, 117, 116, 111, 115, 99, 97, 108, 105, 110, 103, 46, 77, 97, 120, 58, 32, 84, 104, 101, 32, 116, 111, 116, 97, 108, 32, 110, 117, 109, 98, 101, 114, 32, 111, 102, 32, 99, 111, 109, 112, 117, 116, 101, 32, 110, 111, 100, 101, 115, 32, 102, 111, 114, 32, 97, 32, 115, 105, 110, 103, 108, 101, 32, 99, 108, 117, 115, 116, 101, 114, 32, 39, 55, 48, 48, 39, 32, 101, 120, 99, 101, 101, 100, 115, 32, 116, 104, 101, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:134]: Unexpected error:
    <*fmt.wrapError | 0xc000962c60>: 
    failed to create NodePool autoscale-az: failed starting nodepool creation "autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-9ntvp4": PUT https://rp.j6696320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9ntvp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Replicas+Autoscaling.Max: The total number of compute nodes for a single cluster '700' exceeds the maximum allowed '500'. Reduce the total compute nodes requested to be within the maximum allowed."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool autoscale-az: failed starting nodepool creation \"autoscale-az\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9ntvp4\": PUT https://rp.j6696320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9ntvp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Replicas+Autoscaling.Max: The total number of compute nodes for a single cluster '700' exceeds the maximum allowed '500'. Reduce the total compute nodes requested to be within the maximum allowed.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000962c40>{
            msg: "failed starting nodepool creation \"autoscale-az\" for cluster \"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-9ntvp4\": PUT https://rp.j6696320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9ntvp4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Replicas+Autoscaling.Max: The total number of compute nodes for a single cluster '700' exceeds the maximum allowed '500'. Reduce the total compute nodes requested to be within the maximum allowed.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000b967b0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "0db94d04-dd8e-47c4-a152-06aa92aab279",
                        ],
                        "Date": [
                            "Wed, 18 Mar 2026 19:48:49 GMT",
                        ],
                        "Content-Length": ["283"],
                        "X-Envoy-Upstream-Service-Time": ["1179"],
                        "Server": ["istio-envoy"],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000962b80>{
                        s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 82, 101, 112, 108, 105, 99, 97, 115, 43, 65, 117, 116, 111, 115, 99, 97, 108, 105, 110, 103, 46, 77, 97, 120, 58, 32, 84, 104, 101, 32, 116, 111, 116, 97, 108, 32, 110, 117, 109, 98, 101, 114, 32, 111, 102, 32, 99, 111, 109, 112, 117, 116, 101, 32, 110, 111, 100, 101, 115, 32, 102, 111, 114, 32, 97, 32, 115, 105, 110, 103, 108, 101, 32, 99, 108, 117, 115, 116, 101, 114, 32, 39, 55, 48, 48, 39, 32, 101, 120, 99, 101, 101, 100, 115, 32, 116, 104, 101, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
18:32:41 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0xc001620070>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0xc001620070>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9cfdfc7
 n/a e2e failed ★#4484 (closed) 1
e2e (1)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
18:24:25 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc000af7a10>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc000af7a10>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 04f43f0 · merge 04f43f0
 n/a e2e failed ★#4482 (merged) 4
e2e (4)
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
18:12:30 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009fda40>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-dd4h2h" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-dd4h2h, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-dd4h2h\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-dd4h2h, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009e77a0>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-dd4h2h",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009fda40>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-dd4h2h" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-dd4h2h, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-dd4h2h\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-dd4h2h, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009e77a0>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-dd4h2h",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
route host was never found: Get "<url>": dial tcp <ip>:<port>: connect: connection timed out
18:12:30 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc000b8e160>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-pnpwp.apps.aro.x7y4h7y8r4t2i2c.bnid.j2153088.hcp.osadev.cloud": dial tcp 4.236.24.249:443: connect: connection timed out
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-pnpwp.apps.aro.x7y4h7y8r4t2i2c.bnid.j2153088.hcp.osadev.cloud\": dial tcp 4.236.24.249:443: connect: connection timed out",
        err: <*url.Error | 0xc000daa0f0>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-pnpwp.apps.aro.x7y4h7y8r4t2i2c.bnid.j2153088.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc0009ea140>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: <*net.TCPAddr | 0xc000daa300>{
                    IP: [4, 236, 24, 249],
                    Port: 443,
                    Zone: "",
                },
                Err: <*os.SyscallError | 0xc000b8e120>{
                    Syscall: "connect",
                    Err: <syscall.Errno>0x6e,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc000b8e160>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-pnpwp.apps.aro.x7y4h7y8r4t2i2c.bnid.j2153088.hcp.osadev.cloud": dial tcp 4.236.24.249:443: connect: connection timed out
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-pnpwp.apps.aro.x7y4h7y8r4t2i2c.bnid.j2153088.hcp.osadev.cloud\": dial tcp 4.236.24.249:443: connect: connection timed out",
        err: <*url.Error | 0xc000daa0f0>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-pnpwp.apps.aro.x7y4h7y8r4t2i2c.bnid.j2153088.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc0009ea140>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: <*net.TCPAddr | 0xc000daa300>{
                    IP: [4, 236, 24, 249],
                    Port: 443,
                    Zone: "",
                },
                Err: <*os.SyscallError | 0xc000b8e120>{
                    Syscall: "connect",
                    Err: <syscall.Errno>0x6e,
                },
            },
        },
    }
occurred
Interrupted by User
18:12:30 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
18:12:30 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.002s.
Expected success, but got an error:
    <*errors.errorString | 0xc00072aa00>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.002s.
Expected success, but got an error:
    <*errors.errorString | 0xc00072aa00>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head e5f8a5d
 n/a e2e failed ★#4157 (open) 25
e2e (25)
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
18:09:09 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:106]: Unexpected error:
    <*fmt.wrapErrors | 0xc001001b90>: 
    failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-gtlnpg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-gtlnpg, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-gtlnpg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-gtlnpg, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000bf2720>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-gtlnpg",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:106]: Unexpected error:
    <*fmt.wrapErrors | 0xc001001b90>: 
    failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="basic-cluster-gtlnpg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-gtlnpg, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-gtlnpg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-gtlnpg, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000bf2720>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster basic-hcp-cluster in resource group basic-cluster-gtlnpg",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
18:09:09 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc001465590>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-hlgz5x" in resourcegroup="admin-credential-lifecycle-test-j4scpw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-hlgz5x in resource group admin-credential-lifecycle-test-j4scpw, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-hlgz5x\" in resourcegroup=\"admin-credential-lifecycle-test-j4scpw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-hlgz5x in resource group admin-credential-lifecycle-test-j4scpw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011320f0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-hlgz5x in resource group admin-credential-lifecycle-test-j4scpw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc001465590>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-hlgz5x" in resourcegroup="admin-credential-lifecycle-test-j4scpw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-hlgz5x in resource group admin-credential-lifecycle-test-j4scpw, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-hlgz5x\" in resourcegroup=\"admin-credential-lifecycle-test-j4scpw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-hlgz5x in resource group admin-credential-lifecycle-test-j4scpw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011320f0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-hlgz5x in resource group admin-credential-lifecycle-test-j4scpw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
18:09:09 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000f301f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7nkn7p" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0007e6d80>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7nkn7p\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0007e6d60>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7nkn7p\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0007e6d40>{
                        msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7nkn7p\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000f301f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7nkn7p" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0007e6d80>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7nkn7p\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0007e6d60>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7nkn7p\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0007e6d40>{
                        msg: "failed waiting for hcpCluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-7nkn7p\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
18:09:09 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapErrors | 0xc000158c30>: 
    failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-ddvp7b" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-ddvp7b, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-ddvp7b\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-ddvp7b, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000581720>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-ddvp7b",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapErrors | 0xc000158c30>: 
    failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-ddvp7b" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-ddvp7b, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-ddvp7b\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-ddvp7b, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000581720>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-ddvp7b",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
18:09:09 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000f9e330>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-lpr96q" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001616f40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-lpr96q\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001616f20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-lpr96q\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001616f00>{
                        msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-lpr96q\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000f9e330>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-lpr96q" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001616f40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-lpr96q\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001616f20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-lpr96q\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001616f00>{
                        msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-lpr96q\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
18:09:09 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0010ca210>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-w6bnfq" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00094dd40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-w6bnfq\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00094dd20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-w6bnfq\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00094dd00>{
                        msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-w6bnfq\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0010ca210>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-w6bnfq" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00094dd40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-w6bnfq\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00094dd20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-w6bnfq\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00094dd00>{
                        msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-w6bnfq\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
18:09:09 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00020b1b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-4ggzfv" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0008b7920>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-4ggzfv\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0008b7900>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-4ggzfv\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0008b78e0>{
                        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-4ggzfv\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00020b1b8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-4ggzfv" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0008b7920>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-4ggzfv\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0008b7900>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-4ggzfv\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0008b78e0>{
                        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-4ggzfv\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
18:09:09 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:115]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009ba480>: 
    failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-vsqvd2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-vsqvd2, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-vsqvd2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-vsqvd2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e15260>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-vsqvd2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:115]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009ba480>: 
    failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-vsqvd2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-vsqvd2, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-vsqvd2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-vsqvd2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e15260>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-vsqvd2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
18:09:09 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:105]: Unexpected error:
    <*fmt.wrapErrors | 0xc00124a600>: 
    failed waiting for hcpCluster="cluster-zstream-4-20-8tn56p" in resourcegroup="rg-zstream-upgrade-4-20-fhqqrg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-8tn56p in resource group rg-zstream-upgrade-4-20-fhqqrg, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-8tn56p\" in resourcegroup=\"rg-zstream-upgrade-4-20-fhqqrg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-8tn56p in resource group rg-zstream-upgrade-4-20-fhqqrg, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001357d00>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-8tn56p in resource group rg-zstream-upgrade-4-20-fhqqrg",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:105]: Unexpected error:
    <*fmt.wrapErrors | 0xc00124a600>: 
    failed waiting for hcpCluster="cluster-zstream-4-20-8tn56p" in resourcegroup="rg-zstream-upgrade-4-20-fhqqrg" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-8tn56p in resource group rg-zstream-upgrade-4-20-fhqqrg, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-8tn56p\" in resourcegroup=\"rg-zstream-upgrade-4-20-fhqqrg\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-8tn56p in resource group rg-zstream-upgrade-4-20-fhqqrg, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001357d00>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-8tn56p in resource group rg-zstream-upgrade-4-20-fhqqrg",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Deserializaion Error: no output from command
18:09:09 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
s report:
      Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled (Spec Runtime: 1h29m59.817s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
        In [It] (Node Runtime: 1h29m59.817s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
          At [By Step] creating the HCP cluster (Step Runtime: 40m59.377s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:76

          Spec Goroutine
          goroutine 124 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8266018, 0xc000219270}, 0x783ee71?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8299000, {0x8266018, 0xc000219270}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8266088?, 0xc000309650?}, {{0x827b848?, 0xc000ed7300?}, 0xc000a45c30?}, 0xc00111d7e0, {0xc000df20d8, 0x15}, {0x786569b, 0x14}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x781f451, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func24.2({0x7f4080059c60, 0xc000f06ed0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:77
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b920?, 0xc000f06ed0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 121
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83 @ 03/18/26 20:20:42.433
  "ts"="2026-03-18 20:20:42.433473" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 20:20:42.433670" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-xs77kz" "resourceGroup"="np-autoscaling-kmxg9d"
  "ts"="2026-03-18 20:20:43.529045" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-xs77kz" "resourceGroup"=""
  "ts"="2026-03-18 20:20:44.451510" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-18 20:20:44.451656" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-18 20:20:44.451753" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 20:20:44.453327" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-29"
  "ts"="2026-03-18 20:20:44.453420" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-kmxg9d"
  "ts"="2026-03-18 20:20:45.097298" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-29" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 20:20:45.644412" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-kmxg9d" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs"
  "ts"="2026-03-18 20:20:46.080042" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-29" "deployment"="msi-deployment-29"
  "ts"="2026-03-18 20:20:46.547007" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-kmxg9d" "deployment"="customer-infra-np-autoscale-cluster-xs77kz"
  "ts"="2026-03-18 20:20:47.609960" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 20:20:47.610258" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 20:20:47.610426" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-kmxg9d"
Command Error: signal: killed
Deserializaion Error: no output from command
s report:
      Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled (Spec Runtime: 1h29m59.817s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
        In [It] (Node Runtime: 1h29m59.817s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:36
          At [By Step] creating the HCP cluster (Step Runtime: 40m59.377s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:76

          Spec Goroutine
          goroutine 124 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8266018, 0xc000219270}, 0x783ee71?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8299000, {0x8266018, 0xc000219270}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8266088?, 0xc000309650?}, {{0x827b848?, 0xc000ed7300?}, 0xc000a45c30?}, 0xc00111d7e0, {0xc000df20d8, 0x15}, {0x786569b, 0x14}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x781f451, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func24.2({0x7f4080059c60, 0xc000f06ed0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:77
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b920?, 0xc000f06ed0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 121
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83 @ 03/18/26 20:20:42.433
  "ts"="2026-03-18 20:20:42.433473" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 20:20:42.433670" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-autoscale-cluster-xs77kz" "resourceGroup"="np-autoscaling-kmxg9d"
  "ts"="2026-03-18 20:20:43.529045" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-autoscale-cluster-xs77kz" "resourceGroup"=""
  "ts"="2026-03-18 20:20:44.451510" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-18 20:20:44.451656" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-3ace844abea5a4d3a28ffc64801a1d716d83f775e2e8d3dceadd0cb72b88474b.yaml.gz"
  "ts"="2026-03-18 20:20:44.451753" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 20:20:44.453327" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-29"
  "ts"="2026-03-18 20:20:44.453420" "level"=0 "msg"="collecting deployments" "resourceGroup"="np-autoscaling-kmxg9d"
  "ts"="2026-03-18 20:20:45.097298" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-29" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 20:20:45.644412" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-kmxg9d" "deployment"="pooledNonMsiScopedAssignments-ssrwqrnhyyabs"
  "ts"="2026-03-18 20:20:46.080042" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-29" "deployment"="msi-deployment-29"
  "ts"="2026-03-18 20:20:46.547007" "level"=0 "msg"="collecting operations" "resourceGroup"="np-autoscaling-kmxg9d" "deployment"="customer-infra-np-autoscale-cluster-xs77kz"
  "ts"="2026-03-18 20:20:47.609960" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 20:20:47.610258" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 20:20:47.610426" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="np-autoscaling-kmxg9d"
Command Error: signal: killed
Deserializaion Error: no output from command
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
18:09:09 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ffb7d0>: 
    failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-47h67v" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-47h67v, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-47h67v\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-47h67v, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009f5eb0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-47h67v",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ffb7d0>: 
    failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-47h67v" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-47h67v, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-47h67v\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-47h67v, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009f5eb0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-47h67v",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network
18:09:09 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc0009e82c0>: 
    failed to create HCP cluster ea-listvbdptj: failed waiting for cluster="ea-listvbdptj" in resourcegroup="ea-list-8zflz5" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821",
      "name": "01e981d5-e5aa-4ec0-a800-053ac6e07821",
      "status": "Failed",
      "startTime": "2026-03-18T18:51:34.435159653Z",
      "endTime": "2026-03-18T18:54:16.669219148Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-8zflz5/providers/Microsoft.Network/virtualNetworks/customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-listvbdptj: failed waiting for cluster=\"ea-listvbdptj\" in resourcegroup=\"ea-list-8zflz5\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821\",\n  \"name\": \"01e981d5-e5aa-4ec0-a800-053ac6e07821\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:34.435159653Z\",\n  \"endTime\": \"2026-03-18T18:54:16.669219148Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-8zflz5/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009e82a0>{
            msg: "failed waiting for cluster=\"ea-listvbdptj\" in resourcegroup=\"ea-list-8zflz5\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821\",\n  \"name\": \"01e981d5-e5aa-4ec0-a800-053ac6e07821\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:34.435159653Z\",\n  \"endTime\": \"2026-03-18T18:54:16.669219148Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-8zflz5/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc0009e82c0>: 
    failed to create HCP cluster ea-listvbdptj: failed waiting for cluster="ea-listvbdptj" in resourcegroup="ea-list-8zflz5" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821",
      "name": "01e981d5-e5aa-4ec0-a800-053ac6e07821",
      "status": "Failed",
      "startTime": "2026-03-18T18:51:34.435159653Z",
      "endTime": "2026-03-18T18:54:16.669219148Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-8zflz5/providers/Microsoft.Network/virtualNetworks/customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-listvbdptj: failed waiting for cluster=\"ea-listvbdptj\" in resourcegroup=\"ea-list-8zflz5\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821\",\n  \"name\": \"01e981d5-e5aa-4ec0-a800-053ac6e07821\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:34.435159653Z\",\n  \"endTime\": \"2026-03-18T18:54:16.669219148Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-8zflz5/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009e82a0>{
            msg: "failed waiting for cluster=\"ea-listvbdptj\" in resourcegroup=\"ea-list-8zflz5\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01e981d5-e5aa-4ec0-a800-053ac6e07821\",\n  \"name\": \"01e981d5-e5aa-4ec0-a800-053ac6e07821\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:34.435159653Z\",\n  \"endTime\": \"2026-03-18T18:54:16.669219148Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-8zflz5/providers/Microsoft.Network/virtualNetworks/customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
18:09:09 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00158c1e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-pn5hvg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0008b4560>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-pn5hvg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0008b4540>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-pn5hvg\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000c41ee0>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-pn5hvg\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00158c1e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-pn5hvg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0008b4560>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-pn5hvg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0008b4540>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-pn5hvg\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000c41ee0>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-pn5hvg\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
18:09:09 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0015521e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lrrfr6" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d53e40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-lrrfr6\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d53e20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-lrrfr6\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000d53e00>{
                        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-lrrfr6\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0015521e0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lrrfr6" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000d53e40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-lrrfr6\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000d53e20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-lrrfr6\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000d53e00>{
                        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-lrrfr6\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
18:09:09 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s.
Expected success, but got an error:
    <*errors.errorString | 0xc000ac88b0>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.007s.
Expected success, but got an error:
    <*errors.errorString | 0xc000ac88b0>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
18:09:09 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0013e04b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-vw57dt" in resourcegroup="cluster-listing-fh2v8t" to finish deleting: context canceled
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-bbj57t" in resourcegroup="cluster-listing-htksvg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000f056a0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-vw57dt\" in resourcegroup=\"cluster-listing-fh2v8t\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000f05680>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-vw57dt\" in resourcegroup=\"cluster-listing-fh2v8t\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000f05660>{
                        msg: "failed waiting for hcpCluster=\"list-test-cluster-vw57dt\" in resourcegroup=\"cluster-listing-fh2v8t\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xc001602d40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-bbj57t\" in resourcegroup=\"cluster-listing-htksvg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001602d20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-bbj57t\" in resourcegroup=\"cluster-listing-htksvg\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001602d00>{
                        msg: "failed waiting for hcpCluster=\"list-test-cluster-bbj57t\" in resourcegroup=\"cluster-listing-htksvg\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0013e04b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-vw57dt" in resourcegroup="cluster-listing-fh2v8t" to finish deleting: context canceled
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="list-test-cluster-bbj57t" in resourcegroup="cluster-listing-htksvg" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000f056a0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-vw57dt\" in resourcegroup=\"cluster-listing-fh2v8t\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000f05680>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-vw57dt\" in resourcegroup=\"cluster-listing-fh2v8t\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000f05660>{
                        msg: "failed waiting for hcpCluster=\"list-test-cluster-vw57dt\" in resourcegroup=\"cluster-listing-fh2v8t\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
            <*fmt.wrapError | 0xc001602d40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-bbj57t\" in resourcegroup=\"cluster-listing-htksvg\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001602d20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"list-test-cluster-bbj57t\" in resourcegroup=\"cluster-listing-htksvg\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001602d00>{
                        msg: "failed waiting for hcpCluster=\"list-test-cluster-bbj57t\" in resourcegroup=\"cluster-listing-htksvg\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
18:09:09 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:105]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009e0090>: 
    failed waiting for hcpCluster="cluster-zstream-4-21-fz7m8h" in resourcegroup="rg-zstream-upgrade-4-21-9vvm8n" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-fz7m8h in resource group rg-zstream-upgrade-4-21-9vvm8n, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-fz7m8h\" in resourcegroup=\"rg-zstream-upgrade-4-21-9vvm8n\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-fz7m8h in resource group rg-zstream-upgrade-4-21-9vvm8n, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000572810>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-fz7m8h in resource group rg-zstream-upgrade-4-21-9vvm8n",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:105]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009e0090>: 
    failed waiting for hcpCluster="cluster-zstream-4-21-fz7m8h" in resourcegroup="rg-zstream-upgrade-4-21-9vvm8n" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-fz7m8h in resource group rg-zstream-upgrade-4-21-9vvm8n, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-21-fz7m8h\" in resourcegroup=\"rg-zstream-upgrade-4-21-9vvm8n\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-fz7m8h in resource group rg-zstream-upgrade-4-21-9vvm8n, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000572810>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-21-fz7m8h in resource group rg-zstream-upgrade-4-21-9vvm8n",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
18:09:09 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012a2498>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-jc2htq" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000c95760>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-jc2htq\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000c95740>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-jc2htq\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000c95720>{
                        msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-jc2htq\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012a2498>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-jc2htq" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000c95760>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-jc2htq\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000c95740>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-jc2htq\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000c95720>{
                        msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-jc2htq\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
18:09:09 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapErrors | 0xc001119d40>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-c4hrv4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-c4hrv4, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-c4hrv4\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-c4hrv4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009660b0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-c4hrv4",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapErrors | 0xc001119d40>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-c4hrv4" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-c4hrv4, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-c4hrv4\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-c4hrv4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009660b0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-c4hrv4",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network
18:09:09 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapError | 0xc0011a2a00>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-796vrt" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152",
      "name": "f7e35d02-2fd8-4915-a123-3030870ba152",
      "status": "Failed",
      "startTime": "2026-03-18T18:51:34.793318672Z",
      "endTime": "2026-03-18T19:06:34.464660362Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'arm64-vm-customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-796vrt/providers/Microsoft.Network/virtualNetworks/arm64-vm-customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-796vrt\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152\",\n  \"name\": \"f7e35d02-2fd8-4915-a123-3030870ba152\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:34.793318672Z\",\n  \"endTime\": \"2026-03-18T19:06:34.464660362Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'arm64-vm-customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-796vrt/providers/Microsoft.Network/virtualNetworks/arm64-vm-customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0011a29e0>{
            msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-796vrt\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152\",\n  \"name\": \"f7e35d02-2fd8-4915-a123-3030870ba152\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:34.793318672Z\",\n  \"endTime\": \"2026-03-18T19:06:34.464660362Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'arm64-vm-customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-796vrt/providers/Microsoft.Network/virtualNetworks/arm64-vm-customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additiona...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapError | 0xc0011a2a00>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-796vrt" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152",
      "name": "f7e35d02-2fd8-4915-a123-3030870ba152",
      "status": "Failed",
      "startTime": "2026-03-18T18:51:34.793318672Z",
      "endTime": "2026-03-18T19:06:34.464660362Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'arm64-vm-customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-796vrt/providers/Microsoft.Network/virtualNetworks/arm64-vm-customer-vnet\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-796vrt\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152\",\n  \"name\": \"f7e35d02-2fd8-4915-a123-3030870ba152\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:34.793318672Z\",\n  \"endTime\": \"2026-03-18T19:06:34.464660362Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'arm64-vm-customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-796vrt/providers/Microsoft.Network/virtualNetworks/arm64-vm-customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0011a29e0>{
            msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-796vrt\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7e35d02-2fd8-4915-a123-3030870ba152\",\n  \"name\": \"f7e35d02-2fd8-4915-a123-3030870ba152\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:34.793318672Z\",\n  \"endTime\": \"2026-03-18T19:06:34.464660362Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'arm64-vm-customer-vnet' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-796vrt/providers/Microsoft.Network/virtualNetworks/arm64-vm-customer-vnet\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additiona...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Deserializaion Error: no output from command
18:09:09 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel
Show raw failure
re's a current progress report:
      Customer should be able to update node pool labels and taints (Spec Runtime: 1h29m59.927s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
        In [It] (Node Runtime: 1h29m59.927s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
          At [By Step] creating the HCP cluster (Step Runtime: 21m9.913s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:83

          Spec Goroutine
          goroutine 44 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8266018, 0xc000806500}, 0x783ee71?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8299000, {0x8266018, 0xc000806500}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8266088?, 0xc0005ec150?}, {{0x827b848?, 0xc000cbe300?}, 0xc000a971d0?}, 0xc00009b4f0, {0xc001307e20, 0x1a}, {0x78794ae, 0x18}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x781f451, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func25.1({0x7f72e04d1460, 0xc000b19e60})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:84
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(
                | 	ctx,
                | 	GinkgoLogr,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b920?, 0xc000b19e60?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 100
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91 @ 03/18/26 20:20:44.776
  "ts"="2026-03-18 20:20:44.776726" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 20:20:44.776825" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-xqhhfh" "resourceGroup"="rg-np-labels-taints-tgzmm2"
  "ts"="2026-03-18 20:20:45.909847" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-xqhhfh" "resourceGroup"=""
  "ts"="2026-03-18 20:20:46.664043" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-03-18 20:20:46.664206" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-03-18 20:20:46.664290" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 20:20:46.666099" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-60"
  "ts"="2026-03-18 20:20:46.666112" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-tgzmm2"
  "ts"="2026-03-18 20:20:47.286227" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-60" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 20:20:47.883058" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-tgzmm2" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
  "ts"="2026-03-18 20:20:48.248972" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-60" "deployment"="msi-deployment-60"
  "ts"="2026-03-18 20:20:48.769750" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-tgzmm2" "deployment"="customer-infra-cluster-np-labels-taints-xqhhfh"
  "ts"="2026-03-18 20:20:49.765203" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 20:20:49.765412" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 20:20:49.765637" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-tgzmm2"
Command Error: signal: killed
Deserializaion Error: no output from command
re's a current progress report:
      Customer should be able to update node pool labels and taints (Spec Runtime: 1h29m59.927s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
        In [It] (Node Runtime: 1h29m59.927s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
          At [By Step] creating the HCP cluster (Step Runtime: 21m9.913s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:83

          Spec Goroutine
          goroutine 44 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8266018, 0xc000806500}, 0x783ee71?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8299000, {0x8266018, 0xc000806500}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8266088?, 0xc0005ec150?}, {{0x827b848?, 0xc000cbe300?}, 0xc000a971d0?}, 0xc00009b4f0, {0xc001307e20, 0x1a}, {0x78794ae, 0x18}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x781f451, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func25.1({0x7f72e04d1460, 0xc000b19e60})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:84
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(
                | 	ctx,
                | 	GinkgoLogr,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b920?, 0xc000b19e60?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 100
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91 @ 03/18/26 20:20:44.776
  "ts"="2026-03-18 20:20:44.776726" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 20:20:44.776825" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-xqhhfh" "resourceGroup"="rg-np-labels-taints-tgzmm2"
  "ts"="2026-03-18 20:20:45.909847" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-xqhhfh" "resourceGroup"=""
  "ts"="2026-03-18 20:20:46.664043" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-03-18 20:20:46.664206" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
  "ts"="2026-03-18 20:20:46.664290" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 20:20:46.666099" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-60"
  "ts"="2026-03-18 20:20:46.666112" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-tgzmm2"
  "ts"="2026-03-18 20:20:47.286227" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-60" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 20:20:47.883058" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-tgzmm2" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
  "ts"="2026-03-18 20:20:48.248972" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-60" "deployment"="msi-deployment-60"
  "ts"="2026-03-18 20:20:48.769750" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-tgzmm2" "deployment"="customer-infra-cluster-np-labels-taints-xqhhfh"
  "ts"="2026-03-18 20:20:49.765203" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 20:20:49.765412" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 20:20:49.765637" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-tgzmm2"
Command Error: signal: killed
Deserializaion Error: no output from command
Deserializaion Error: no output from command
18:09:09 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
stomer should be able to update nodepool replicas and autoscaling (Spec Runtime: 1h29m59.937s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:35
        In [It] (Node Runtime: 1h29m59.937s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:35
          At [By Step] creating the HCP cluster (Step Runtime: 21m20.215s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:75

          Spec Goroutine
          goroutine 96 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8266018, 0xc0007e4640}, 0x783ee71?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8299000, {0x8266018, 0xc0007e4640}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8266088?, 0xc0006d22a0?}, {{0x827b848?, 0xc000814500?}, 0xc001606a30?}, 0xc001287600, {0xc000b40320, 0x1c}, {0x7888076, 0x1b}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x781f451, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func26.1({0x7f9f24269e60, 0xc000960f90})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:76
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b920?, 0xc000960f90?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 94
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82 @ 03/18/26 20:20:45.017
  "ts"="2026-03-18 20:20:45.017585" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 20:20:45.017722" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-update-nodes-hcp-cluster-d7c8dd" "resourceGroup"="nodepool-update-nodes-nj695s"
  "ts"="2026-03-18 20:20:46.051413" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-update-nodes-hcp-cluster-d7c8dd" "resourceGroup"=""
  "ts"="2026-03-18 20:20:46.935614" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4d7111da7878873ab737ec2a993f733d74239d4c919cccd4b38c36a47c0a2edd.yaml.gz"
  "ts"="2026-03-18 20:20:46.935743" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4d7111da7878873ab737ec2a993f733d74239d4c919cccd4b38c36a47c0a2edd.yaml.gz"
  "ts"="2026-03-18 20:20:46.935831" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 20:20:46.937287" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-62"
  "ts"="2026-03-18 20:20:46.937308" "level"=0 "msg"="collecting deployments" "resourceGroup"="nodepool-update-nodes-nj695s"
  "ts"="2026-03-18 20:20:47.628469" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-62" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 20:20:48.065445" "level"=0 "msg"="collecting operations" "resourceGroup"="nodepool-update-nodes-nj695s" "deployment"="pooledNonMsiScopedAssignments-w3k44uxsb3qey"
  "ts"="2026-03-18 20:20:48.541601" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-62" "deployment"="msi-deployment-62"
  "ts"="2026-03-18 20:20:49.112141" "level"=0 "msg"="collecting operations" "resourceGroup"="nodepool-update-nodes-nj695s" "deployment"="customer-infra-np-update-nodes-hcp-cluster-d7c8dd"
  "ts"="2026-03-18 20:20:50.054507" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 20:20:50.054711" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 20:20:50.054902" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="nodepool-update-nodes-nj695s"
Command Error: signal: killed
Deserializaion Error: no output from command
stomer should be able to update nodepool replicas and autoscaling (Spec Runtime: 1h29m59.937s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:35
        In [It] (Node Runtime: 1h29m59.937s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:35
          At [By Step] creating the HCP cluster (Step Runtime: 21m20.215s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:75

          Spec Goroutine
          goroutine 96 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8266018, 0xc0007e4640}, 0x783ee71?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8299000, {0x8266018, 0xc0007e4640}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8266088?, 0xc0006d22a0?}, {{0x827b848?, 0xc000814500?}, 0xc001606a30?}, 0xc001287600, {0xc000b40320, 0x1c}, {0x7888076, 0x1b}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x781f451, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func26.1({0x7f9f24269e60, 0xc000960f90})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:76
                | 
                | By("creating the HCP cluster")
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b920?, 0xc000960f90?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 94
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82 @ 03/18/26 20:20:45.017
  "ts"="2026-03-18 20:20:45.017585" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 20:20:45.017722" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-update-nodes-hcp-cluster-d7c8dd" "resourceGroup"="nodepool-update-nodes-nj695s"
  "ts"="2026-03-18 20:20:46.051413" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-update-nodes-hcp-cluster-d7c8dd" "resourceGroup"=""
  "ts"="2026-03-18 20:20:46.935614" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-4d7111da7878873ab737ec2a993f733d74239d4c919cccd4b38c36a47c0a2edd.yaml.gz"
  "ts"="2026-03-18 20:20:46.935743" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-4d7111da7878873ab737ec2a993f733d74239d4c919cccd4b38c36a47c0a2edd.yaml.gz"
  "ts"="2026-03-18 20:20:46.935831" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 20:20:46.937287" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-62"
  "ts"="2026-03-18 20:20:46.937308" "level"=0 "msg"="collecting deployments" "resourceGroup"="nodepool-update-nodes-nj695s"
  "ts"="2026-03-18 20:20:47.628469" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-62" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 20:20:48.065445" "level"=0 "msg"="collecting operations" "resourceGroup"="nodepool-update-nodes-nj695s" "deployment"="pooledNonMsiScopedAssignments-w3k44uxsb3qey"
  "ts"="2026-03-18 20:20:48.541601" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-62" "deployment"="msi-deployment-62"
  "ts"="2026-03-18 20:20:49.112141" "level"=0 "msg"="collecting operations" "resourceGroup"="nodepool-update-nodes-nj695s" "deployment"="customer-infra-np-update-nodes-hcp-cluster-d7c8dd"
  "ts"="2026-03-18 20:20:50.054507" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 20:20:50.054711" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 20:20:50.054902" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="nodepool-update-nodes-nj695s"
Command Error: signal: killed
Deserializaion Error: no output from command
Deserializaion Error: no output from command
18:09:09 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
si-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
          > github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000ab4b98, {0xc000c7f860?, 0x0?, 0x0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
          > github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000bdea88, {0x0, 0x0}, {0xb38f160, 0x0, 0x0}, {0xc00007c064, 0x2a}, 0x0, {0xc000c7f840, ...})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
          > github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007c064, 0x2a}, ...)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
            github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2a5da65?, 0xc000785ec0?})
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/ginkgo/util.go:85
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x8266018, 0xc000c645a0}, 0xc000533e00, 0x1)
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:596
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:237
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230

          goroutine 24 [select]
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000533e00?)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 87
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95 @ 03/18/26 20:20:42.018
  "ts"="2026-03-18 20:20:42.018432" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 20:20:42.018540" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-zstream-4-19-vxcbt8-vxf8pq" "resourceGroup"="rg-zstream-upgrade-4-19-bck7b7"
  "ts"="2026-03-18 20:20:43.160580" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-zstream-4-19-vxcbt8-vxf8pq" "resourceGroup"=""
  "ts"="2026-03-18 20:20:43.898952" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-0c1d721d40414cddab3a52b9d7add0203efdd55a4c816fc13c12379212f5cea0.yaml.gz"
  "ts"="2026-03-18 20:20:43.899124" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-0c1d721d40414cddab3a52b9d7add0203efdd55a4c816fc13c12379212f5cea0.yaml.gz"
  "ts"="2026-03-18 20:20:43.899218" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 20:20:43.900635" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-100"
  "ts"="2026-03-18 20:20:43.900698" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-zstream-upgrade-4-19-bck7b7"
  "ts"="2026-03-18 20:20:44.490292" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-100" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 20:20:45.076813" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-19-bck7b7" "deployment"="pooledNonMsiScopedAssignments-mq76aepq5vgly"
  "ts"="2026-03-18 20:20:45.542169" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-100" "deployment"="msi-deployment-100"
  "ts"="2026-03-18 20:20:46.026274" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-19-bck7b7" "deployment"="customer-infra-cluster-zstream-4-19-vxcbt8-vxf8pq"
  "ts"="2026-03-18 20:20:46.979701" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 20:20:46.979869" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 20:20:46.980038" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-zstream-upgrade-4-19-bck7b7"
Command Error: signal: killed
Deserializaion Error: no output from command
si-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:199
          > github.com/onsi/ginkgo/v2/internal.(*group).run(0xc000ab4b98, {0xc000c7f860?, 0x0?, 0x0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349
          > github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000bdea88, {0x0, 0x0}, {0xb38f160, 0x0, 0x0}, {0xc00007c064, 0x2a}, 0x0, {0xc000c7f840, ...})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496
          > github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007c064, 0x2a}, ...)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68
            github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2a5da65?, 0xc000785ec0?})
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/ginkgo/util.go:85
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x8266018, 0xc000c645a0}, 0xc000533e00, 0x1)
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:596
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:237
            github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
              /go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230

          goroutine 24 [select]
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts.func2(0xc000533e00?)
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:131
          > github.com/onsi/ginkgo/v2/internal/interrupt_handler.(*InterruptHandler).registerForInterrupts in goroutine 87
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/interrupt_handler/interrupt_handler.go:128
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95 @ 03/18/26 20:20:42.018
  "ts"="2026-03-18 20:20:42.018432" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 20:20:42.018540" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-zstream-4-19-vxcbt8-vxf8pq" "resourceGroup"="rg-zstream-upgrade-4-19-bck7b7"
  "ts"="2026-03-18 20:20:43.160580" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-zstream-4-19-vxcbt8-vxf8pq" "resourceGroup"=""
  "ts"="2026-03-18 20:20:43.898952" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-0c1d721d40414cddab3a52b9d7add0203efdd55a4c816fc13c12379212f5cea0.yaml.gz"
  "ts"="2026-03-18 20:20:43.899124" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-0c1d721d40414cddab3a52b9d7add0203efdd55a4c816fc13c12379212f5cea0.yaml.gz"
  "ts"="2026-03-18 20:20:43.899218" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 20:20:43.900635" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-100"
  "ts"="2026-03-18 20:20:43.900698" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-zstream-upgrade-4-19-bck7b7"
  "ts"="2026-03-18 20:20:44.490292" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-100" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 20:20:45.076813" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-19-bck7b7" "deployment"="pooledNonMsiScopedAssignments-mq76aepq5vgly"
  "ts"="2026-03-18 20:20:45.542169" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-100" "deployment"="msi-deployment-100"
  "ts"="2026-03-18 20:20:46.026274" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-zstream-upgrade-4-19-bck7b7" "deployment"="customer-infra-cluster-zstream-4-19-vxcbt8-vxf8pq"
  "ts"="2026-03-18 20:20:46.979701" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 20:20:46.979869" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 20:20:46.980038" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-zstream-upgrade-4-19-bck7b7"
Command Error: signal: killed
Deserializaion Error: no output from command
Deserializaion Error: no output from command
18:09:09 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
progress report:
      Customer should respect cluster-wide node limits with nodepool autoscaling (Spec Runtime: 1h29m59.735s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122
        In [It] (Node Runtime: 1h29m59.735s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122
          At [By Step] creating cluster with MaxNodesTotal limit (Step Runtime: 41m56.808s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:143

          Spec Goroutine
          goroutine 96 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8266018, 0xc000e9b040}, 0x783ee71?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8299000, {0x8266018, 0xc000e9b040}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8266088?, 0xc000ac4d90?}, {{0x827b848?, 0xc000e95880?}, 0xc001680d20?}, 0xc0010ef810, {0xc000aeb230, 0x16}, {0x785ae5f, 0x12}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x781f451, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func24.3({0x7f676f6cc2a0, 0xc000ea3ef0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:160
                | Expect(err).NotTo(HaveOccurred())
                | 
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b920?, 0xc000ea3ef0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 93
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166 @ 03/18/26 20:20:42.506
  "ts"="2026-03-18 20:20:42.507185" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 20:20:42.507302" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-node-limit-cluster-q2pqxn" "resourceGroup"="node-limit-test-dbckdp"
  "ts"="2026-03-18 20:20:43.677122" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-node-limit-cluster-q2pqxn" "resourceGroup"=""
  "ts"="2026-03-18 20:20:44.490455" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz"
  "ts"="2026-03-18 20:20:44.490632" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz"
  "ts"="2026-03-18 20:20:44.490759" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 20:20:44.493288" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-148"
  "ts"="2026-03-18 20:20:44.493345" "level"=0 "msg"="collecting deployments" "resourceGroup"="node-limit-test-dbckdp"
  "ts"="2026-03-18 20:20:45.137419" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-148" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 20:20:45.646482" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-dbckdp" "deployment"="pooledNonMsiScopedAssignments-j6nw3ohx7bzo4"
  "ts"="2026-03-18 20:20:46.128381" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-148" "deployment"="msi-deployment-148"
  "ts"="2026-03-18 20:20:46.632160" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-dbckdp" "deployment"="customer-infra-node-limit-cluster-q2pqxn"
  "ts"="2026-03-18 20:20:47.650775" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 20:20:47.650925" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 20:20:47.651309" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="node-limit-test-dbckdp"
Command Error: signal: killed
Deserializaion Error: no output from command
progress report:
      Customer should respect cluster-wide node limits with nodepool autoscaling (Spec Runtime: 1h29m59.735s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122
        In [It] (Node Runtime: 1h29m59.735s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122
          At [By Step] creating cluster with MaxNodesTotal limit (Step Runtime: 41m56.808s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:143

          Spec Goroutine
          goroutine 96 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x8266018, 0xc000e9b040}, 0x783ee71?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x8299000, {0x8266018, 0xc000e9b040}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x8266088?, 0xc000ac4d90?}, {{0x827b848?, 0xc000e95880?}, 0xc001680d20?}, 0xc0010ef810, {0xc000aeb230, 0x16}, {0x785ae5f, 0x12}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:658
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x781f451, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func24.3({0x7f676f6cc2a0, 0xc000ea3ef0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:160
                | Expect(err).NotTo(HaveOccurred())
                | 
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b920?, 0xc000ea3ef0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 93
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166 @ 03/18/26 20:20:42.506
  "ts"="2026-03-18 20:20:42.507185" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 20:20:42.507302" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-node-limit-cluster-q2pqxn" "resourceGroup"="node-limit-test-dbckdp"
  "ts"="2026-03-18 20:20:43.677122" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-node-limit-cluster-q2pqxn" "resourceGroup"=""
  "ts"="2026-03-18 20:20:44.490455" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz"
  "ts"="2026-03-18 20:20:44.490632" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz"
  "ts"="2026-03-18 20:20:44.490759" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 20:20:44.493288" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-148"
  "ts"="2026-03-18 20:20:44.493345" "level"=0 "msg"="collecting deployments" "resourceGroup"="node-limit-test-dbckdp"
  "ts"="2026-03-18 20:20:45.137419" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-148" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 20:20:45.646482" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-dbckdp" "deployment"="pooledNonMsiScopedAssignments-j6nw3ohx7bzo4"
  "ts"="2026-03-18 20:20:46.128381" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-148" "deployment"="msi-deployment-148"
  "ts"="2026-03-18 20:20:46.632160" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-dbckdp" "deployment"="customer-infra-node-limit-cluster-q2pqxn"
  "ts"="2026-03-18 20:20:47.650775" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 20:20:47.650925" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 20:20:47.651309" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="node-limit-test-dbckdp"
Command Error: signal: killed
Deserializaion Error: no output from command
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network
18:09:09 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapError | 0xc0016e5560>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-njw9zk" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69",
      "name": "ffea3a4c-4428-4297-adbc-cee4edb4dc69",
      "status": "Failed",
      "startTime": "2026-03-18T18:51:36.043041035Z",
      "endTime": "2026-03-18T18:52:41.369770418Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-njw9zk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-njw9zk\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69\",\n  \"name\": \"ffea3a4c-4428-4297-adbc-cee4edb4dc69\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:36.043041035Z\",\n  \"endTime\": \"2026-03-18T18:52:41.369770418Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-njw9zk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0016e5540>{
            msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-njw9zk\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69\",\n  \"name\": \"ffea3a4c-4428-4297-adbc-cee4edb4dc69\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:36.043041035Z\",\n  \"endTime\": \"2026-03-18T18:52:41.369770418Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-njw9zk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapError | 0xc0016e5560>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-njw9zk" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69",
      "name": "ffea3a4c-4428-4297-adbc-cee4edb4dc69",
      "status": "Failed",
      "startTime": "2026-03-18T18:51:36.043041035Z",
      "endTime": "2026-03-18T18:52:41.369770418Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-njw9zk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-njw9zk\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69\",\n  \"name\": \"ffea3a4c-4428-4297-adbc-cee4edb4dc69\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:36.043041035Z\",\n  \"endTime\": \"2026-03-18T18:52:41.369770418Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-njw9zk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0016e5540>{
            msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-njw9zk\" to finish creating: GET https://rp.j4399616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffea3a4c-4428-4297-adbc-cee4edb4dc69\",\n  \"name\": \"ffea3a4c-4428-4297-adbc-cee4edb4dc69\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T18:51:36.043041035Z\",\n  \"endTime\": \"2026-03-18T18:52:41.369770418Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-njw9zk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 64a2f7a · merge 855eea8
 n/a e2e failed ★#4425 (merged) 3
e2e (3)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
17:58:52 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0xc00108f610>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0xc00108f610>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
17:58:52 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ffa228>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-tbxqkj" in resourcegroup="admin-credential-lifecycle-test-j5ckd5" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00173af40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-tbxqkj\" in resourcegroup=\"admin-credential-lifecycle-test-j5ckd5\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00173af20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-tbxqkj\" in resourcegroup=\"admin-credential-lifecycle-test-j5ckd5\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00173af00>{
                        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-tbxqkj\" in resourcegroup=\"admin-credential-lifecycle-test-j5ckd5\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb1af0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000ffa228>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-tbxqkj" in resourcegroup="admin-credential-lifecycle-test-j5ckd5" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00173af40>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-tbxqkj\" in resourcegroup=\"admin-credential-lifecycle-test-j5ckd5\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00173af20>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"admin-cred-lifecycle-tbxqkj\" in resourcegroup=\"admin-credential-lifecycle-test-j5ckd5\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00173af00>{
                        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-tbxqkj\" in resourcegroup=\"admin-credential-lifecycle-test-j5ckd5\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb1af0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
Interrupted by User
17:58:52 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 3e72d40
 n/a e2e failed ★#4516 (open) 1
e2e (1)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
17:53:53 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc00140a820>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc00140a820>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c84eed0 · merge 9feee09
 n/a e2e failed ★#4004 (merged) 2
e2e (2)
Deserializaion Error: no output from command
17:45:42 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
upResourceGroups.func1({0x7f93243730a0, 0xc001695e60})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:294 +0x14d fp=0xc000b77fc0 sp=0xc000b77f28 pc=0x4ec572d
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:303 +0x28 fp=0xc000b77fe0 sp=0xc000b77fc0 pc=0x4ec55a8
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000b77fe8 sp=0xc000b77fe0 pc=0x2ad7921
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 1224
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:287 +0xc5

goroutine 1449 gp=0xc000e03180 m=nil [IO wait]:
runtime.gopark(0x0?, 0x2424ab48?, 0x50?, 0xf7?, 0xb?)
	/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc0000cf720 sp=0xc0000cf700 pc=0x2acf28e
runtime.netpollblock(0x2b38d58?, 0x2a5b046?, 0x0?)
	/usr/local/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc0000cf758 sp=0xc0000cf720 pc=0x2a8ee97
internal/poll.runtime_pollWait(0x7f9327dbc000, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0x85 fp=0xc0000cf778 sp=0xc0000cf758 pc=0x2acdb25
internal/poll.(*pollDesc).wait(0xc000294700?, 0xc000e1a000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000cf7a0 sp=0xc0000cf778 pc=0x2b5ba87
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000294700, {0xc000e1a000, 0x5500, 0x5500})
	/usr/local/go/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc0000cf838 sp=0xc0000cf7a0 pc=0x2b5ceb9
net.(*netFD).Read(0xc000294700, {0xc000e1a000?, 0xc0000cf948?, 0x2a5895c?})
	/usr/local/go/src/net/fd_posix.go:68 +0x25 fp=0xc0000cf880 sp=0xc0000cf838 pc=0x2cf1d65
net.(*conn).Read(0xc000ef60a0, {0xc000e1a000?, 0x7f932436f9e8?, 0x7f936e9a3a78?})
	/usr/local/go/src/net/net.go:196 +0x45 fp=0xc0000cf8c8 sp=0xc0000cf880 pc=0x2d03e05
crypto/tls.(*atLeastReader).Read(0xc00104e3d8, {0xc000e1a000?, 0x784a078?, 0xe?})
	/usr/local/go/src/crypto/tls/conn.go:819 +0x3b fp=0xc0000cf910 sp=0xc0000cf8c8 pc=0x2d6f6bb
bytes.(*Buffer).ReadFrom(0xc00091e2a8, {0x81cfac0, 0xc00104e3d8})
	/usr/local/go/src/bytes/buffer.go:217 +0x98 fp=0xc0000cf968 sp=0xc0000cf910 pc=0x2b99a38
crypto/tls.(*Conn).readFromUntil(0xc00091e008, {0x81d04c0, 0xc000ef60a0}, 0xc0000cf9d0?)
	/usr/local/go/src/crypto/tls/conn.go:841 +0xde fp=0xc0000cf9a0 sp=0xc0000cf968 pc=0x2d6f89e
crypto/tls.(*Conn).readRecordOrCCS(0xc00091e008, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:630 +0x3db fp=0xc0000cfc18 sp=0xc0000cf9a0 pc=0x2d6c91b
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc00091e008, {0xc000934000, 0x1000, 0x31aed60?})
	/usr/local/go/src/crypto/tls/conn.go:1397 +0x145 fp=0xc0000cfc80 sp=0xc0000cfc18 pc=0x2d73205
bufio.(*Reader).Read(0xc001751b60, {0xc0014442e4, 0x9, 0x31bce0e?})
	/usr/local/go/src/bufio/bufio.go:245 +0x197 fp=0xc0000cfcb8 sp=0xc0000cfc80 pc=0x2c351f7
io.ReadAtLeast({0x81cf0a0, 0xc001751b60}, {0xc0014442e4, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:335 +0x8e fp=0xc0000cfd00 sp=0xc0000cfcb8 pc=0x2b29aee
io.ReadFull(...)
	/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0014442e4, 0x9, 0xc000000011?}, {0x81cf0a0?, 0xc001751b60?})
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc0000cfd50 sp=0xc0000cfd00 pc=0x31a5b25
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0014442a0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc0000cfdd8 sp=0xc0000cfd50 pc=0x31a66eb
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0014442a0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc0000cfe18 sp=0xc0000cfdd8 pc=0x31a6e58
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0000cffa8)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc0000cff60 sp=0xc0000cfe18 pc=0x31b9d4a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000d048c0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc0000cffc8 sp=0xc0000cff60 pc=0x31b90d2
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc0000cffe0 sp=0xc0000cffc8 pc=0x31b2865
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000cffe8 sp=0xc0000cffe0 pc=0x2ad7921
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 1448
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5

rax    0xca
rbx    0x0
rcx    0x2ad9743
rdx    0x0
rdi    0xb367af0
rsi    0x80
rbp    0x7ffde5778ed0
rsp    0x7ffde5778e88
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc0000a7208
r13    0x1
r14    0xb365a00
r15    0x1
rip    0x2ad9741
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
upResourceGroups.func1({0x7f93243730a0, 0xc001695e60})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:294 +0x14d fp=0xc000b77fc0 sp=0xc000b77f28 pc=0x4ec572d
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:303 +0x28 fp=0xc000b77fe0 sp=0xc000b77fc0 pc=0x4ec55a8
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000b77fe8 sp=0xc000b77fe0 pc=0x2ad7921
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 1224
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:287 +0xc5

goroutine 1449 gp=0xc000e03180 m=nil [IO wait]:
runtime.gopark(0x0?, 0x2424ab48?, 0x50?, 0xf7?, 0xb?)
	/usr/local/go/src/runtime/proc.go:460 +0xce fp=0xc0000cf720 sp=0xc0000cf700 pc=0x2acf28e
runtime.netpollblock(0x2b38d58?, 0x2a5b046?, 0x0?)
	/usr/local/go/src/runtime/netpoll.go:575 +0xf7 fp=0xc0000cf758 sp=0xc0000cf720 pc=0x2a8ee97
internal/poll.runtime_pollWait(0x7f9327dbc000, 0x72)
	/usr/local/go/src/runtime/netpoll.go:351 +0x85 fp=0xc0000cf778 sp=0xc0000cf758 pc=0x2acdb25
internal/poll.(*pollDesc).wait(0xc000294700?, 0xc000e1a000?, 0x0)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000cf7a0 sp=0xc0000cf778 pc=0x2b5ba87
internal/poll.(*pollDesc).waitRead(...)
	/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000294700, {0xc000e1a000, 0x5500, 0x5500})
	/usr/local/go/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc0000cf838 sp=0xc0000cf7a0 pc=0x2b5ceb9
net.(*netFD).Read(0xc000294700, {0xc000e1a000?, 0xc0000cf948?, 0x2a5895c?})
	/usr/local/go/src/net/fd_posix.go:68 +0x25 fp=0xc0000cf880 sp=0xc0000cf838 pc=0x2cf1d65
net.(*conn).Read(0xc000ef60a0, {0xc000e1a000?, 0x7f932436f9e8?, 0x7f936e9a3a78?})
	/usr/local/go/src/net/net.go:196 +0x45 fp=0xc0000cf8c8 sp=0xc0000cf880 pc=0x2d03e05
crypto/tls.(*atLeastReader).Read(0xc00104e3d8, {0xc000e1a000?, 0x784a078?, 0xe?})
	/usr/local/go/src/crypto/tls/conn.go:819 +0x3b fp=0xc0000cf910 sp=0xc0000cf8c8 pc=0x2d6f6bb
bytes.(*Buffer).ReadFrom(0xc00091e2a8, {0x81cfac0, 0xc00104e3d8})
	/usr/local/go/src/bytes/buffer.go:217 +0x98 fp=0xc0000cf968 sp=0xc0000cf910 pc=0x2b99a38
crypto/tls.(*Conn).readFromUntil(0xc00091e008, {0x81d04c0, 0xc000ef60a0}, 0xc0000cf9d0?)
	/usr/local/go/src/crypto/tls/conn.go:841 +0xde fp=0xc0000cf9a0 sp=0xc0000cf968 pc=0x2d6f89e
crypto/tls.(*Conn).readRecordOrCCS(0xc00091e008, 0x0)
	/usr/local/go/src/crypto/tls/conn.go:630 +0x3db fp=0xc0000cfc18 sp=0xc0000cf9a0 pc=0x2d6c91b
crypto/tls.(*Conn).readRecord(...)
	/usr/local/go/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc00091e008, {0xc000934000, 0x1000, 0x31aed60?})
	/usr/local/go/src/crypto/tls/conn.go:1397 +0x145 fp=0xc0000cfc80 sp=0xc0000cfc18 pc=0x2d73205
bufio.(*Reader).Read(0xc001751b60, {0xc0014442e4, 0x9, 0x31bce0e?})
	/usr/local/go/src/bufio/bufio.go:245 +0x197 fp=0xc0000cfcb8 sp=0xc0000cfc80 pc=0x2c351f7
io.ReadAtLeast({0x81cf0a0, 0xc001751b60}, {0xc0014442e4, 0x9, 0x9}, 0x9)
	/usr/local/go/src/io/io.go:335 +0x8e fp=0xc0000cfd00 sp=0xc0000cfcb8 pc=0x2b29aee
io.ReadFull(...)
	/usr/local/go/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0014442e4, 0x9, 0xc000000011?}, {0x81cf0a0?, 0xc001751b60?})
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc0000cfd50 sp=0xc0000cfd00 pc=0x31a5b25
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0014442a0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc0000cfdd8 sp=0xc0000cfd50 pc=0x31a66eb
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0014442a0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc0000cfe18 sp=0xc0000cfdd8 pc=0x31a6e58
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc0000cffa8)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc0000cff60 sp=0xc0000cfe18 pc=0x31b9d4a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc000d048c0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc0000cffc8 sp=0xc0000cff60 pc=0x31b90d2
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc0000cffe0 sp=0xc0000cffc8 pc=0x31b2865
runtime.goexit({})
	/usr/local/go/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000cffe8 sp=0xc0000cffe0 pc=0x2ad7921
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 1448
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5

rax    0xca
rbx    0x0
rcx    0x2ad9743
rdx    0x0
rdi    0xb367af0
rsi    0x80
rbp    0x7ffde5778ed0
rsp    0x7ffde5778e88
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc0000a7208
r13    0x1
r14    0xb365a00
r15    0x1
rip    0x2ad9741
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
17:45:42 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc00020d290>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc00020d290>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 500db44 · merge e9ca15d
 n/a unknown failed #3907 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head fd99db3
 n/a e2e failed ⚠#4515 (closed) 3
e2e (3)
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
17:07:12 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc00057ccc0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-fclq4l" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-fclq4l, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-fclq4l\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-fclq4l, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0002fae20>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-fclq4l",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:207]: Unexpected error:
    <*fmt.wrapErrors | 0xc00057ccc0>: 
    failed waiting for nodepool="np-one-node" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-fclq4l" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-fclq4l, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-one-node\" in cluster=\"np-update-nodes-hcp-cluster\" resourcegroup=\"nodepool-update-nodes-fclq4l\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-fclq4l, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0002fae20>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-one-node in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-fclq4l",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Customer should be able to use workload identity via the cluster OIDC issuer URL: Timed out after <duration>s.
17:07:12 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:276]: Timed out after 300.007s.
Expected
    <v1.PodPhase>: Running
to equal
    <v1.PodPhase>: Succeeded
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:276]: Timed out after 300.007s.
Expected
    <v1.PodPhase>: Running
to equal
    <v1.PodPhase>: Succeeded
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
17:07:12 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc0008a7e90>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc0008a7e90>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head f89aa70 · merge 61e704d
 n/a e2e failed ★#4514 (merged) 3
e2e (3)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
16:46:07 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0017c61b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-p84c9b" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001663640>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-p84c9b\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001663620>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-p84c9b\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001663600>{
                        msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-p84c9b\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0017c61b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-p84c9b" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc001663640>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-p84c9b\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc001663620>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-p84c9b\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc001663600>{
                        msg: "failed waiting for hcpCluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-p84c9b\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
16:46:07 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s.
Expected success, but got an error:
    <*errors.errorString | 0xc0002c1610>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.006s.
Expected success, but got an error:
    <*errors.errorString | 0xc0002c1610>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
16:46:07 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000012618>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-zwswk7" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0008e1600>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-zwswk7\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0008e15e0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-zwswk7\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0008e15c0>{
                        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-zwswk7\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000012618>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-zwswk7" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0008e1600>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-zwswk7\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0008e15e0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-zwswk7\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc0008e15c0>{
                        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-zwswk7\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0928008 · merge 0928008
 n/a e2e failed ★#4444 (merged) 1
e2e (1)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
16:31:47 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc000e5a2f0>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc000e5a2f0>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ca7bbc9 · merge ca7bbc9
 n/a e2e failed ★#4512 (merged) 2
e2e (2)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
16:13:01 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0xc001177610>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0xc001177610>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network
16:13:01 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapError | 0xc00099e900>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-sj46dk" to finish creating: GET https://rp.j3199232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f",
      "name": "621a39ba-9923-42f5-b1ef-878422819b7f",
      "status": "Failed",
      "startTime": "2026-03-18T16:39:27.380883412Z",
      "endTime": "2026-03-18T16:41:27.012428471Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-sj46dk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-sj46dk\" to finish creating: GET https://rp.j3199232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f\",\n  \"name\": \"621a39ba-9923-42f5-b1ef-878422819b7f\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:39:27.380883412Z\",\n  \"endTime\": \"2026-03-18T16:41:27.012428471Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-sj46dk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00099e8e0>{
            msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-sj46dk\" to finish creating: GET https://rp.j3199232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f\",\n  \"name\": \"621a39ba-9923-42f5-b1ef-878422819b7f\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:39:27.380883412Z\",\n  \"endTime\": \"2026-03-18T16:41:27.012428471Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-sj46dk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapError | 0xc00099e900>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-sj46dk" to finish creating: GET https://rp.j3199232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f",
      "name": "621a39ba-9923-42f5-b1ef-878422819b7f",
      "status": "Failed",
      "startTime": "2026-03-18T16:39:27.380883412Z",
      "endTime": "2026-03-18T16:41:27.012428471Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-sj46dk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-sj46dk\" to finish creating: GET https://rp.j3199232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f\",\n  \"name\": \"621a39ba-9923-42f5-b1ef-878422819b7f\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:39:27.380883412Z\",\n  \"endTime\": \"2026-03-18T16:41:27.012428471Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-sj46dk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00099e8e0>{
            msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-sj46dk\" to finish creating: GET https://rp.j3199232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621a39ba-9923-42f5-b1ef-878422819b7f\",\n  \"name\": \"621a39ba-9923-42f5-b1ef-878422819b7f\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:39:27.380883412Z\",\n  \"endTime\": \"2026-03-18T16:41:27.012428471Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-sj46dk/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 39d25fe · merge 39d25fe
 n/a e2e failed ★#4511 (merged) 1
e2e (1)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
16:12:42 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0xc0009d2200>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0xc0009d2200>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 63bb89f · merge 56bf314
 n/a e2e failed ★#4485 (merged) 2
e2e (2)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
16:12:28 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc000b17ba0>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.008s.
Expected success, but got an error:
    <*errors.errorString | 0xc000b17ba0>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
16:12:28 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015b32f0>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-knf62b" in resourcegroup="admin-credential-lifecycle-test-xjdb4r" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-knf62b in resource group admin-credential-lifecycle-test-xjdb4r, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-knf62b\" in resourcegroup=\"admin-credential-lifecycle-test-xjdb4r\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-knf62b in resource group admin-credential-lifecycle-test-xjdb4r, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001151760>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-knf62b in resource group admin-credential-lifecycle-test-xjdb4r",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc0015b32f0>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-knf62b" in resourcegroup="admin-credential-lifecycle-test-xjdb4r" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-knf62b in resource group admin-credential-lifecycle-test-xjdb4r, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-knf62b\" in resourcegroup=\"admin-credential-lifecycle-test-xjdb4r\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-knf62b in resource group admin-credential-lifecycle-test-xjdb4r, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001151760>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-knf62b in resource group admin-credential-lifecycle-test-xjdb4r",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 04f43f0 · merge 04f43f0
 n/a e2e failed ★#4482 (merged) 2
e2e (2)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
15:48:51 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.003s.
Expected success, but got an error:
    <*errors.errorString | 0xc0015d0e30>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.003s.
Expected success, but got an error:
    <*errors.errorString | 0xc0015d0e30>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network
15:48:51 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc000492920>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-vrcl27" to finish creating: GET https://rp.j6925184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971",
      "name": "116083cc-ad76-4a20-a638-573bdc265971",
      "status": "Failed",
      "startTime": "2026-03-18T16:29:11.853285095Z",
      "endTime": "2026-03-18T16:30:16.704331608Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-vrcl27/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vrcl27\" to finish creating: GET https://rp.j6925184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971\",\n  \"name\": \"116083cc-ad76-4a20-a638-573bdc265971\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:29:11.853285095Z\",\n  \"endTime\": \"2026-03-18T16:30:16.704331608Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-vrcl27/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000492900>{
            msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vrcl27\" to finish creating: GET https://rp.j6925184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971\",\n  \"name\": \"116083cc-ad76-4a20-a638-573bdc265971\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:29:11.853285095Z\",\n  \"endTime\": \"2026-03-18T16:30:16.704331608Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-vrcl27/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' excee...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc000492920>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-vrcl27" to finish creating: GET https://rp.j6925184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971",
      "name": "116083cc-ad76-4a20-a638-573bdc265971",
      "status": "Failed",
      "startTime": "2026-03-18T16:29:11.853285095Z",
      "endTime": "2026-03-18T16:30:16.704331608Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-vrcl27/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vrcl27\" to finish creating: GET https://rp.j6925184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971\",\n  \"name\": \"116083cc-ad76-4a20-a638-573bdc265971\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:29:11.853285095Z\",\n  \"endTime\": \"2026-03-18T16:30:16.704331608Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-vrcl27/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000492900>{
            msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-vrcl27\" to finish creating: GET https://rp.j6925184.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/116083cc-ad76-4a20-a638-573bdc265971\",\n  \"name\": \"116083cc-ad76-4a20-a638-573bdc265971\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:29:11.853285095Z\",\n  \"endTime\": \"2026-03-18T16:30:16.704331608Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-vrcl27/providers/Microsoft.Network/virtualNetworks/customer-vnet-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' excee...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 16b44df · merge 3a9b5c9
 n/a e2e failed ★#4503 (merged) 2
e2e (2)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network
15:37:59 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0000f4e60>: 
    failed to create HCP cluster cluster-zstream-4-21-hbbshx: failed waiting for cluster="cluster-zstream-4-21-hbbshx" in resourcegroup="rg-zstream-upgrade-4-21-nwl58q" to finish creating: GET https://rp.j7523840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c",
      "name": "e6336c4e-2460-4b7d-90ff-a42ce9ed477c",
      "status": "Failed",
      "startTime": "2026-03-18T16:34:11.735754512Z",
      "endTime": "2026-03-18T16:35:04.329397808Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-zstream-hbbshx' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-nwl58q/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hbbshx\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-hbbshx: failed waiting for cluster=\"cluster-zstream-4-21-hbbshx\" in resourcegroup=\"rg-zstream-upgrade-4-21-nwl58q\" to finish creating: GET https://rp.j7523840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c\",\n  \"name\": \"e6336c4e-2460-4b7d-90ff-a42ce9ed477c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:34:11.735754512Z\",\n  \"endTime\": \"2026-03-18T16:35:04.329397808Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-hbbshx' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-nwl58q/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hbbshx\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0000f4e20>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-21-hbbshx\" in resourcegroup=\"rg-zstream-upgrade-4-21-nwl58q\" to finish creating: GET https://rp.j7523840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c\",\n  \"name\": \"e6336c4e-2460-4b7d-90ff-a42ce9ed477c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:34:11.735754512Z\",\n  \"endTime\": \"2026-03-18T16:35:04.329397808Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-hbbshx' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-nwl58q/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hbbshx\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0000f4e60>: 
    failed to create HCP cluster cluster-zstream-4-21-hbbshx: failed waiting for cluster="cluster-zstream-4-21-hbbshx" in resourcegroup="rg-zstream-upgrade-4-21-nwl58q" to finish creating: GET https://rp.j7523840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c",
      "name": "e6336c4e-2460-4b7d-90ff-a42ce9ed477c",
      "status": "Failed",
      "startTime": "2026-03-18T16:34:11.735754512Z",
      "endTime": "2026-03-18T16:35:04.329397808Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-zstream-hbbshx' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-nwl58q/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hbbshx\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-hbbshx: failed waiting for cluster=\"cluster-zstream-4-21-hbbshx\" in resourcegroup=\"rg-zstream-upgrade-4-21-nwl58q\" to finish creating: GET https://rp.j7523840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c\",\n  \"name\": \"e6336c4e-2460-4b7d-90ff-a42ce9ed477c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:34:11.735754512Z\",\n  \"endTime\": \"2026-03-18T16:35:04.329397808Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-hbbshx' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-nwl58q/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hbbshx\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0000f4e20>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-21-hbbshx\" in resourcegroup=\"rg-zstream-upgrade-4-21-nwl58q\" to finish creating: GET https://rp.j7523840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e6336c4e-2460-4b7d-90ff-a42ce9ed477c\",\n  \"name\": \"e6336c4e-2460-4b7d-90ff-a42ce9ed477c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T16:34:11.735754512Z\",\n  \"endTime\": \"2026-03-18T16:35:04.329397808Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-hbbshx' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-nwl58q/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hbbshx\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
15:37:59 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc00082ca60>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc00082ca60>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head be491ea
 n/a unknown failed #4513 (open) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head b6f3e3f · merge 9feee09
 n/a provision failed #4004 (merged) 1
provision (1)
Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "<url>": proxy error from localhost:9443 while dialing <ip>:<port>, code 500: 500 Internal Server Error
15:32:32 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SessionGate/service/deploy · suite step graph
Show raw failure
time=2026-03-18T16:17:09.247Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-18T16:17:09.949Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:f4d4cd29-749f-4add-af25-814804090dfc cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j6301440] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j6301440.hcpsvc.osadev.cloud] image:map[digest:sha256:ee85d3b105f0865d7e9a8e12a07b194acc4b3daccabc436a5ce0811237e0ed1d registry:arohcpsvcdev.azurecr.io repository:ci-op-m0pldgmm/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-m0pldgmm/pipeline:pull workloadIdentityClientId:19499bc9-c8d8-416f-b45f-de8c45af77a8 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:c9a9304b-45b5-4b61-800e-b4ab2499bbdd workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]"
time=2026-03-18T16:17:09.949Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T16:17:10.215Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate
time=2026-03-18T16:17:10.461Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false
time=2026-03-18T16:17:10.461Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T16:17:10.567Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T16:17:17.274Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.23:15017, code 500: 500 Internal Server Error\nInternal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.23:15017, code 500: 500 Internal Server Error\nInternal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.20:15017, code 500: 500 Internal Server Error"
time=2026-03-18T16:17:17.274Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T16:17:17.274Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T16:17:19.116Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy release=sessiongate namespace=sessiongate status=failed description="Release \"sessiongate\" failed: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.23:15017, code 500: 500 Internal Server Error\nInternal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.23:15017, code 500: 500 Internal Server Error\nInternal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.20:15017, code 500: 500 Internal Server Error" values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:f4d4cd29-749f-4add-af25-814804090dfc cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j6301440] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j6301440.hcpsvc.osadev.cloud] image:map[digest:sha256:ee85d3b105f0865d7e9a8e12a07b194acc4b3daccabc436a5ce0811237e0ed1d registry:arohcpsvcdev.azurecr.io repository:ci-op-m0pldgmm/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-m0pldgmm/pipeline:pull workloadIdentityClientId:19499bc9-c8d8-416f-b45f-de8c45af77a8 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:c9a9304b-45b5-4b61-800e-b4ab2499bbdd workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]"
time=2026-03-18T16:17:19.119Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/5RU0W7aMBR9z1dc5alIJCqd1E5MfWDtpEmTJsR46rQHxzkEF8c3unZAVHz85FLajAJlb/bJucc+Pjc3sQgk8NyKhqdbKlVQQRUWF1w8QocfxpVDH8S4qk9b6Keq8QphCfeM+EbpHdz7nRARpb+gBWEsvDQl5M4q79M+pR7eG3aVCsi8biKkFj4zPhjOjKsEkbeTkKXRGGnNrQt71e+225oJW3w1rjSu2j9OswvC1kKy4iDj/zQsVAnJYKGDYfeB5j0ay+sa5/o4ys80JJiZ0XEtmAn8HHLyHUdaxq21HUtNa+0HF75rfeB68tIe95gZZ6LPDt3nnbpcCWdz3eTqqRXkmuvIfFWzrQ+Q+LLvumC7zmrlVIWd424Wx0M4EeD5yR0Reem+8+IatWHOYp5UFByzNXr9HIm1vNoP5aDA9+l0POH2wAH/bpM/X5JFW0AcAvy3+Af6ZEOrOQQUTA0fVN1QgbACHF2UKiDCF+nV5dV1dvkpG3yeDq6Hg5vh4PIh7VGe0ynOzUPa6yUbemTjaGFceWucg3TmBjt6GxfdObE/IJINNcLx69tF+9RwuSWfJ9Iny1WfanivqijJUkKoWHfMl/D6bwAAAP//ZRy9YN0EAAA="
time=2026-03-18T16:17:19.122Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T16:17:19.122Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.23:15017, code 500: 500 Internal Server Error\nInternal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.23:15017, code 500: 500 Internal Server Error\nInternal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": proxy error from localhost:9443 while dialing 10.128.64.20:15017, code 500: 500 Internal Server Error"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a0ece52 · merge a0ece52
 n/a e2e failed ★#4481 (merged) 1
e2e (1)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
15:29:54 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.009s.
Expected success, but got an error:
    <*errors.errorString | 0xc00013cb00>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.009s.
Expected success, but got an error:
    <*errors.errorString | 0xc00013cb00>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ca7bbc9 · merge ca7bbc9
 n/a provision failed ★#4512 (merged) 1
provision (1)
failed to apply namespace aro-hcp-mitigations: Unauthorized
15:27:19 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/mitigations · suite step graph
Show raw failure
time=2026-03-18T15:59:18.908Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations description="Step mitigations\n Kind: Helm\n"
time=2026-03-18T15:59:19.611Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations values=map[recycleOperatorImage:mcr.microsoft.com/aks/command/runtime@sha256:5d1fc3770795996bc22512c28adf4623ecb8a69eec3018a88e34a87bcba3366a]
time=2026-03-18T15:59:19.611Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations
time=2026-03-18T15:59:21.027Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations err="error running Helm release deployment Step, failed to deploy helm release: failed to apply namespace aro-hcp-mitigations: Unauthorized"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 8889208 · merge ec8c17a
 n/a e2e failed ★#4505 (merged) 1
e2e (1)
missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
15:05:43 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc0017197e0>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:94]: Timed out after 600.004s.
Expected success, but got an error:
    <*errors.errorString | 0xc0017197e0>: 
    missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]
    {
        s: "missing expected log sources: [aro-hcp/aro-hcp-frontend aro-hcp/aro-hcp-backend clusters-service/clusters-service-server ocm-arohcp/kube-apiserver]",
    }
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f10a9bd
 n/a provision failed ⚠#4305 (closed) 1
provision (1)
Internal error occurred: failed calling webhook "validation.istio.io": failed to call webhook: Post "<url>": EOF
15:03:47 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SessionGate/service/deploy · suite step graph
Show raw failure
time=2026-03-18T15:43:02.354Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-03-18T15:43:03.013Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:efe1ce4c-67c2-4802-b1e0-c57342ec18bb cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j0351488] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j0351488.hcpsvc.osadev.cloud] image:map[digest:sha256:4437daa84475dc7a0f0b27f08058965c2a7fddf473ee218fbcd9c222cc39de3a registry:arohcpsvcdev.azurecr.io repository:ci-op-5sqnbcdx/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-5sqnbcdx/pipeline:pull workloadIdentityClientId:33911860-31dc-4d70-a529-3868092d1fee workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:c123548d-4a78-4621-8238-b82193881ef2 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]"
time=2026-03-18T15:43:03.014Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T15:43:03.249Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy namespace=sessiongate
time=2026-03-18T15:43:03.594Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy dryRun=false
time=2026-03-18T15:43:03.594Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T15:43:03.722Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T15:43:12.309Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF"
time=2026-03-18T15:43:12.309Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T15:43:12.309Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T15:43:14.366Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy release=sessiongate namespace=sessiongate status=failed description="Release \"sessiongate\" failed: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF" values="map[affinity:map[] auth:map[audience:6dae42f8-4368-4678-94ff-3960e28e3630 authority:https://sts.windows.net/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[clientId:efe1ce4c-67c2-4802-b1e0-c57342ec18bb cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:sessiongate-cert-prow-j0351488] deployment:map[replicas:2 requests:map[cpu:250m memory:512Mi]] httpRoute:map[gateway:map[name:ops-ingress-gateway namespace:aks-istio-ingress sectionName:sessiongate-https] host:sessiongate.j0351488.hcpsvc.osadev.cloud] image:map[digest:sha256:4437daa84475dc7a0f0b27f08058965c2a7fddf473ee218fbcd9c222cc39de3a registry:arohcpsvcdev.azurecr.io repository:ci-op-5sqnbcdx/pipeline] leaderElection:map[leaseDuration:15s renewDeadline:10s retryPeriod:2s] logLevel:2 mise:map[deploy:false policyLabel:Session Gate] nodeSelector:map[] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-5sqnbcdx/pipeline:pull workloadIdentityClientId:33911860-31dc-4d70-a529-3868092d1fee workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] service:map[name:sessiongate port:8080] serviceAccount:map[name:sessiongate workloadIdentityClientId:c123548d-4a78-4621-8238-b82193881ef2 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] tolerations:[]]"
time=2026-03-18T15:43:14.370Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6RUT2vbThC961MMOsWgFfnz+5XikoObFAqFYhyfUnpYrcbSxqsddXbWxsEfvsh2GqPasUNv4unNm515w0scCjAGimwwwC2UWrTowuEFFU9o5Jv15TAIW19lsIW+6wb/QLhAv0FCq80LPPiRAACkdy4GQZ6QwzSDNGAIlnylBdXuWzXa6wob9NIx0mxbeKjCkBcm55B7f84rUg51iazQoRFL/ojI1+l0PKEofaVjPfFXxCCjKDV6sUYfkFa8JSkdpT6i84CGUcZMC1si3zkdwl8bM20H6XlQNoglZX3F2PF2y45BqJnsvLzHmfW295qQ7wnmmknVps31c2TMDTV9Bz5bX1pfvbHT4iDjfRo9X05oPiAvrDnTnXtsHa1ejusf+Mogi511/nZ2zhhDvb3Do3aMDI+jc3vzt9G5E9N1d0RsnzdnNCZnzWrTxDla9tu8X8CT1Cc3OzKG4smFJT8/JfNYIHsUDF+6EAjJGpY1MoLYBoPopoUCZYno4aLUgh18kV5fXn9Qlzfq6uP06v/hfzfDy5vHdAB5Dm9wrq4f08EgWcMTWQ9z68tb6z3yXnSRh9fE2o+qfkYla2iZur+vD82gpXJLPk8kA0dVBg2GoKtOkrhEhmK1N3yJwfwOAAD//7ZNzUpgBQAA"
time=2026-03-18T15:43:14.372Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy
time=2026-03-18T15:43:14.373Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SessionGate resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: Internal error occurred: failed calling webhook \"validation.istio.io\": failed to call webhook: Post \"https://istiod-asm-1-26.aks-istio-system.svc:443/validate?timeout=10s\": EOF"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 16b44df · merge 3a9b5c9
 n/a unknown failed #4503 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ef00dbb · merge ef00dbb
 n/a e2e failed ★#4497 (merged) 1
e2e (1)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network
14:55:05 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapError | 0xc0009409a0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-wxwvnp" to finish creating: GET https://rp.j5514240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0",
      "name": "70020686-e4ff-4c0e-b027-4850cd3b68c0",
      "status": "Failed",
      "startTime": "2026-03-18T15:40:16.05219438Z",
      "endTime": "2026-03-18T15:42:12.275796995Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-wxwvnp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wxwvnp\" to finish creating: GET https://rp.j5514240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0\",\n  \"name\": \"70020686-e4ff-4c0e-b027-4850cd3b68c0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:40:16.05219438Z\",\n  \"endTime\": \"2026-03-18T15:42:12.275796995Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-wxwvnp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000940960>{
            msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wxwvnp\" to finish creating: GET https://rp.j5514240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0\",\n  \"name\": \"70020686-e4ff-4c0e-b027-4850cd3b68c0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:40:16.05219438Z\",\n  \"endTime\": \"2026-03-18T15:42:12.275796995Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-wxwvnp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again afte...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapError | 0xc0009409a0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-wxwvnp" to finish creating: GET https://rp.j5514240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0",
      "name": "70020686-e4ff-4c0e-b027-4850cd3b68c0",
      "status": "Failed",
      "startTime": "2026-03-18T15:40:16.05219438Z",
      "endTime": "2026-03-18T15:42:12.275796995Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-wxwvnp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wxwvnp\" to finish creating: GET https://rp.j5514240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0\",\n  \"name\": \"70020686-e4ff-4c0e-b027-4850cd3b68c0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:40:16.05219438Z\",\n  \"endTime\": \"2026-03-18T15:42:12.275796995Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-wxwvnp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000940960>{
            msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-wxwvnp\" to finish creating: GET https://rp.j5514240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70020686-e4ff-4c0e-b027-4850cd3b68c0\",\n  \"name\": \"70020686-e4ff-4c0e-b027-4850cd3b68c0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:40:16.05219438Z\",\n  \"endTime\": \"2026-03-18T15:42:12.275796995Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-wxwvnp/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again afte...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head c9de0da
 n/a e2e failed ⚠#4477 (closed) 5
e2e (5)
ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity
14:50:04 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:135]: Unexpected error:
    <*exported.ResponseError | 0xc000acd110>: 
    GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f2b4c9f-393b-475e-872c-13718d862016
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f2b4c9f-393b-475e-872c-13718d862016",
      "name": "1f2b4c9f-393b-475e-872c-13718d862016",
      "status": "Failed",
      "startTime": "2026-03-18T15:30:37.846077324Z",
      "endTime": "2026-03-18T15:32:04.274089786Z",
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "Cluster provisioning failed due to multiple errors",
        "details": [
          {
            "code": "InternalServerError",
            "message": "Failed to get virtual network 'customer-vnet2-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h4qth6/providers/Microsoft.Network/virtualNetworks/customer-vnet2-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
          },
          {
            "code": "InternalServerError",
            "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "MultipleErrorsOccurred",
        StatusCode: 200,
        RawResponse: {
            Status: "200 OK",
            StatusCode: 200,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Type": ["application/json"],
                "X-Ms-Request-Id": [
                    "469e3d3c-dab5-4186-9f1b-626c37a65092",
                ],
                "Date": [
                    "Wed, 18 Mar 2026 15:39:21 GMT",
                ],
                "X-Envoy-Upstream-Service-Time": ["7"],
                "Server": ["istio-envoy"],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000b08400>{
                s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 49, 102, 50, 98, 52, 99, 57, 102, 45, 51, 57, 51, 98, 45, 52, 55, 53, 101, 45, 56, 55, 50, 99, 45, 49, 51, 55, 49, 56, 100, 56, 54, 50, 48, 49, 54, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 49, 102, 50, 98, 52, 99, 57, 102, 45, 51, 57, 51, 98, 45, 52, 55, 53, 101, 45, 56, 55, 50, 99, 45, 49, 51, 55, 49, 56, 100, 56, 54, 50, 48, 49, 54, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 48, 58, 51, 55, 46, 56, 52, 54, 48, 55, 55, 51, 50, 52, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 50, 58, 48, 52, 46, 50, 55, 52, 48, 56, 57, 55, 56, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 32, 102, 97, 105, 108, 101, 100, 32, 100, 117, 101, 32, 116, 111, 32, 109, 117, 108, 116, 105, 112, 108, 101, 32, 101, 114, 114, 111, 114, 115, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 100, 101, 116, 97, 105, 108, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 118, 105, 114, 116, 117, 97, 108, 32, 110, 101, 116, 119, 111, 114, 107, 32, 39, 99, 117, 115, 116, 111, 109, 101, 114, 45, 118, 110, 101, 116, 50, 45, 110, 97, 109, 101, 39, 32, 100, 101, 116, 97, 105, 108, 115, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:135]: Unexpected error:
    <*exported.ResponseError | 0xc000acd110>: 
    GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f2b4c9f-393b-475e-872c-13718d862016
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f2b4c9f-393b-475e-872c-13718d862016",
      "name": "1f2b4c9f-393b-475e-872c-13718d862016",
      "status": "Failed",
      "startTime": "2026-03-18T15:30:37.846077324Z",
      "endTime": "2026-03-18T15:32:04.274089786Z",
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "Cluster provisioning failed due to multiple errors",
        "details": [
          {
            "code": "InternalServerError",
            "message": "Failed to get virtual network 'customer-vnet2-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-h4qth6/providers/Microsoft.Network/virtualNetworks/customer-vnet2-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
          },
          {
            "code": "InternalServerError",
            "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "MultipleErrorsOccurred",
        StatusCode: 200,
        RawResponse: {
            Status: "200 OK",
            StatusCode: 200,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Type": ["application/json"],
                "X-Ms-Request-Id": [
                    "469e3d3c-dab5-4186-9f1b-626c37a65092",
                ],
                "Date": [
                    "Wed, 18 Mar 2026 15:39:21 GMT",
                ],
                "X-Envoy-Upstream-Service-Time": ["7"],
                "Server": ["istio-envoy"],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000b08400>{
                s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 49, 102, 50, 98, 52, 99, 57, 102, 45, 51, 57, 51, 98, 45, 52, 55, 53, 101, 45, 56, 55, 50, 99, 45, 49, 51, 55, 49, 56, 100, 56, 54, 50, 48, 49, 54, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 49, 102, 50, 98, 52, 99, 57, 102, 45, 51, 57, 51, 98, 45, 52, 55, 53, 101, 45, 56, 55, 50, 99, 45, 49, 51, 55, 49, 56, 100, 56, 54, 50, 48, 49, 54, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 48, 58, 51, 55, 46, 56, 52, 54, 48, 55, 55, 51, 50, 52, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 50, 58, 48, 52, 46, 50, 55, 52, 48, 56, 57, 55, 56, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 77, 117, 108, 116, 105, 112, 108, 101, 69, 114, 114, 111, 114, 115, 79, 99, 99, 117, 114, 114, 101, 100, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 67, 108, 117, 115, 116, 101, 114, 32, 112, 114, 111, 118, 105, 115, 105, 111, 110, 105, 110, 103, 32, 102, 97, 105, 108, 101, 100, 32, 100, 117, 101, 32, 116, 111, 32, 109, 117, 108, 116, 105, 112, 108, 101, 32, 101, 114, 114, 111, 114, 115, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 100, 101, 116, 97, 105, 108, 115, 34, 58, 32, 91, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 118, 105, 114, 116, 117, 97, 108, 32, 110, 101, 116, 119, 111, 114, 107, 32, 39, 99, 117, 115, 116, 111, 109, 101, 114, 45, 118, 110, 101, 116, 50, 45, 110, 97, 109, 101, 39, 32, 100, 101, 116, 97, 105, 108, 115, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity
14:50:04 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapError | 0xc0007aa280>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-b2rgrg" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5",
      "name": "0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5",
      "status": "Failed",
      "startTime": "2026-03-18T15:53:21.008761573Z",
      "endTime": "2026-03-18T15:54:00.900097609Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-b2rgrg\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n  \"name\": \"0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:53:21.008761573Z\",\n  \"endTime\": \"2026-03-18T15:54:00.900097609Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007aa260>{
            msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-b2rgrg\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n  \"name\": \"0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:53:21.008761573Z\",\n  \"endTime\": \"2026-03-18T15:54:00.900097609Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------------------------------...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapError | 0xc0007aa280>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-b2rgrg" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5",
      "name": "0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5",
      "status": "Failed",
      "startTime": "2026-03-18T15:53:21.008761573Z",
      "endTime": "2026-03-18T15:54:00.900097609Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-b2rgrg\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n  \"name\": \"0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:53:21.008761573Z\",\n  \"endTime\": \"2026-03-18T15:54:00.900097609Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007aa260>{
            msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-b2rgrg\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n  \"name\": \"0b0ba2cf-bb21-4b22-8217-c5b79c26b1e5\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:53:21.008761573Z\",\n  \"endTime\": \"2026-03-18T15:54:00.900097609Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-186/providers/Microsoft.ManagedIdentity/userAssignedIdentities/image-registry\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n----------------------------------------------------...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Interrupted by User
14:50:04 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network
14:50:04 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapError | 0xc0007bea60>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-rm29ks" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5",
      "name": "a598f117-d124-44d5-93ef-ffae6f5ba2e5",
      "status": "Failed",
      "startTime": "2026-03-18T15:29:40.838949406Z",
      "endTime": "2026-03-18T15:31:03.468090691Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rm29ks/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rm29ks\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5\",\n  \"name\": \"a598f117-d124-44d5-93ef-ffae6f5ba2e5\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:29:40.838949406Z\",\n  \"endTime\": \"2026-03-18T15:31:03.468090691Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rm29ks/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007bea40>{
            msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rm29ks\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5\",\n  \"name\": \"a598f117-d124-44d5-93ef-ffae6f5ba2e5\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:29:40.838949406Z\",\n  \"endTime\": \"2026-03-18T15:31:03.468090691Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rm29ks/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapError | 0xc0007bea60>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-rm29ks" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5",
      "name": "a598f117-d124-44d5-93ef-ffae6f5ba2e5",
      "status": "Failed",
      "startTime": "2026-03-18T15:29:40.838949406Z",
      "endTime": "2026-03-18T15:31:03.468090691Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rm29ks/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rm29ks\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5\",\n  \"name\": \"a598f117-d124-44d5-93ef-ffae6f5ba2e5\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:29:40.838949406Z\",\n  \"endTime\": \"2026-03-18T15:31:03.468090691Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rm29ks/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007bea40>{
            msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rm29ks\" to finish creating: GET https://rp.j4968960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a598f117-d124-44d5-93ef-ffae6f5ba2e5\",\n  \"name\": \"a598f117-d124-44d5-93ef-ffae6f5ba2e5\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:29:40.838949406Z\",\n  \"endTime\": \"2026-03-18T15:31:03.468090691Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rm29ks/providers/Microsoft.Network/networkSecurityGroups/customer-nsg\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
14:50:04 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc000b23080>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-dcl9tx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-dcl9tx, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-dcl9tx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-dcl9tx, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00117f460>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-dcl9tx",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc000b23080>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-dcl9tx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-dcl9tx, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-dcl9tx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-dcl9tx, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00117f460>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-dcl9tx",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 5a75b4f · merge 5a75b4f
 n/a e2e failed ★#4491 (merged) 1
e2e (1)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network
14:49:24 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc00075b500>: 
    failed to create HCP cluster cluster-zstream-4-21-hjll2f: failed waiting for cluster="cluster-zstream-4-21-hjll2f" in resourcegroup="rg-zstream-upgrade-4-21-v8v2zt" to finish creating: GET https://rp.j4369792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e",
      "name": "09d881ad-ded7-4498-8e1a-ea625d67d91e",
      "status": "Failed",
      "startTime": "2026-03-18T15:39:54.750498141Z",
      "endTime": "2026-03-18T15:42:18.68414891Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-zstream-hjll2f' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-v8v2zt/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hjll2f\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-hjll2f: failed waiting for cluster=\"cluster-zstream-4-21-hjll2f\" in resourcegroup=\"rg-zstream-upgrade-4-21-v8v2zt\" to finish creating: GET https://rp.j4369792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e\",\n  \"name\": \"09d881ad-ded7-4498-8e1a-ea625d67d91e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:39:54.750498141Z\",\n  \"endTime\": \"2026-03-18T15:42:18.68414891Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-hjll2f' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-v8v2zt/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hjll2f\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00075b4c0>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-21-hjll2f\" in resourcegroup=\"rg-zstream-upgrade-4-21-v8v2zt\" to finish creating: GET https://rp.j4369792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e\",\n  \"name\": \"09d881ad-ded7-4498-8e1a-ea625d67d91e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:39:54.750498141Z\",\n  \"endTime\": \"2026-03-18T15:42:18.68414891Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-hjll2f' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-v8v2zt/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hjll2f\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a68...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc00075b500>: 
    failed to create HCP cluster cluster-zstream-4-21-hjll2f: failed waiting for cluster="cluster-zstream-4-21-hjll2f" in resourcegroup="rg-zstream-upgrade-4-21-v8v2zt" to finish creating: GET https://rp.j4369792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e",
      "name": "09d881ad-ded7-4498-8e1a-ea625d67d91e",
      "status": "Failed",
      "startTime": "2026-03-18T15:39:54.750498141Z",
      "endTime": "2026-03-18T15:42:18.68414891Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-zstream-hjll2f' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-v8v2zt/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hjll2f\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-hjll2f: failed waiting for cluster=\"cluster-zstream-4-21-hjll2f\" in resourcegroup=\"rg-zstream-upgrade-4-21-v8v2zt\" to finish creating: GET https://rp.j4369792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e\",\n  \"name\": \"09d881ad-ded7-4498-8e1a-ea625d67d91e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:39:54.750498141Z\",\n  \"endTime\": \"2026-03-18T15:42:18.68414891Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-hjll2f' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-v8v2zt/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hjll2f\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00075b4c0>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-21-hjll2f\" in resourcegroup=\"rg-zstream-upgrade-4-21-v8v2zt\" to finish creating: GET https://rp.j4369792.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/09d881ad-ded7-4498-8e1a-ea625d67d91e\",\n  \"name\": \"09d881ad-ded7-4498-8e1a-ea625d67d91e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:39:54.750498141Z\",\n  \"endTime\": \"2026-03-18T15:42:18.68414891Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-zstream-hjll2f' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-v8v2zt/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-hjll2f\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a68...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 8550ece · merge 8550ece
 n/a n/a passed #4219 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 47d7e02 · merge f9b0dd4
 n/a e2e failed ★#4257 (merged) 3
e2e (3)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity
14:38:52 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error:
    <*exported.ResponseError | 0xc0011f6d50>: 
    GET https://rp.j9095680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3df1fad-5b33-490c-8720-40caecee25af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3df1fad-5b33-490c-8720-40caecee25af",
      "name": "c3df1fad-5b33-490c-8720-40caecee25af",
      "status": "Failed",
      "startTime": "2026-03-18T15:37:10.011477634Z",
      "endTime": "2026-03-18T15:37:57.291321656Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InternalServerError",
        StatusCode: 200,
        RawResponse: {
            Status: "200 OK",
            StatusCode: 200,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Type": ["application/json"],
                "X-Ms-Request-Id": [
                    "1f44e88a-3cf3-4e30-9689-672c60f0dee9",
                ],
                "Date": [
                    "Wed, 18 Mar 2026 15:38:00 GMT",
                ],
                "Content-Length": ["1561"],
                "X-Envoy-Upstream-Service-Time": ["6"],
                "Server": ["istio-envoy"],
            },
            Body: <*exported.nopClosingBytesReader | 0xc0004c6c60>{
                s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 99, 51, 100, 102, 49, 102, 97, 100, 45, 53, 98, 51, 51, 45, 52, 57, 48, 99, 45, 56, 55, 50, 48, 45, 52, 48, 99, 97, 101, 99, 101, 101, 50, 53, 97, 102, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 99, 51, 100, 102, 49, 102, 97, 100, 45, 53, 98, 51, 51, 45, 52, 57, 48, 99, 45, 56, 55, 50, 48, 45, 52, 48, 99, 97, 101, 99, 101, 101, 50, 53, 97, 102, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 55, 58, 49, 48, 46, 48, 49, 49, 52, 55, 55, 54, 51, 52, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 55, 58, 53, 55, 46, 50, 57, 49, 51, 50, 49, 54, 53, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 50, 50, 51, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129]: Unexpected error:
    <*exported.ResponseError | 0xc0011f6d50>: 
    GET https://rp.j9095680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3df1fad-5b33-490c-8720-40caecee25af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3df1fad-5b33-490c-8720-40caecee25af",
      "name": "c3df1fad-5b33-490c-8720-40caecee25af",
      "status": "Failed",
      "startTime": "2026-03-18T15:37:10.011477634Z",
      "endTime": "2026-03-18T15:37:57.291321656Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-223/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InternalServerError",
        StatusCode: 200,
        RawResponse: {
            Status: "200 OK",
            StatusCode: 200,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Type": ["application/json"],
                "X-Ms-Request-Id": [
                    "1f44e88a-3cf3-4e30-9689-672c60f0dee9",
                ],
                "Date": [
                    "Wed, 18 Mar 2026 15:38:00 GMT",
                ],
                "Content-Length": ["1561"],
                "X-Envoy-Upstream-Service-Time": ["6"],
                "Server": ["istio-envoy"],
            },
            Body: <*exported.nopClosingBytesReader | 0xc0004c6c60>{
                s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 99, 51, 100, 102, 49, 102, 97, 100, 45, 53, 98, 51, 51, 45, 52, 57, 48, 99, 45, 56, 55, 50, 48, 45, 52, 48, 99, 97, 101, 99, 101, 101, 50, 53, 97, 102, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 99, 51, 100, 102, 49, 102, 97, 100, 45, 53, 98, 51, 51, 45, 52, 57, 48, 99, 45, 56, 55, 50, 48, 45, 52, 48, 99, 97, 101, 99, 101, 101, 50, 53, 97, 102, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 55, 58, 49, 48, 46, 48, 49, 49, 52, 55, 55, 54, 51, 52, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 51, 55, 58, 53, 55, 46, 50, 57, 49, 51, 50, 49, 54, 53, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 50, 50, 51, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 99, 111, 110, 116, 114, 111, 108, 45, 112, 108, 97, 110, 101, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
route host was never found: Get "<url>": dial tcp <ip>:<port>: connect: connection timed out
14:38:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc0015e40e0>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-p8q8d.apps.aro.z3v9u1z0j7n5w0w.fdfh.j9095680.hcp.osadev.cloud": dial tcp 51.57.113.19:443: connect: connection timed out
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-p8q8d.apps.aro.z3v9u1z0j7n5w0w.fdfh.j9095680.hcp.osadev.cloud\": dial tcp 51.57.113.19:443: connect: connection timed out",
        err: <*url.Error | 0xc000a48540>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-p8q8d.apps.aro.z3v9u1z0j7n5w0w.fdfh.j9095680.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc000d9e000>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: <*net.TCPAddr | 0xc000a489c0>{
                    IP: [51, 57, 113, 19],
                    Port: 443,
                    Zone: "",
                },
                Err: <*os.SyscallError | 0xc0015e40a0>{
                    Syscall: "connect",
                    Err: <syscall.Errno>0x6e,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:137]: Unexpected error:
    <*fmt.wrapError | 0xc0015e40e0>: 
    route host was never found: Get "https://agnhost-e2e-serving-app-p8q8d.apps.aro.z3v9u1z0j7n5w0w.fdfh.j9095680.hcp.osadev.cloud": dial tcp 51.57.113.19:443: connect: connection timed out
    {
        msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-p8q8d.apps.aro.z3v9u1z0j7n5w0w.fdfh.j9095680.hcp.osadev.cloud\": dial tcp 51.57.113.19:443: connect: connection timed out",
        err: <*url.Error | 0xc000a48540>{
            Op: "Get",
            URL: "https://agnhost-e2e-serving-app-p8q8d.apps.aro.z3v9u1z0j7n5w0w.fdfh.j9095680.hcp.osadev.cloud",
            Err: <*net.OpError | 0xc000d9e000>{
                Op: "dial",
                Net: "tcp",
                Source: nil,
                Addr: <*net.TCPAddr | 0xc000a489c0>{
                    IP: [51, 57, 113, 19],
                    Port: 443,
                    Zone: "",
                },
                Err: <*os.SyscallError | 0xc0015e40a0>{
                    Syscall: "connect",
                    Err: <syscall.Errno>0x6e,
                },
            },
        },
    }
occurred
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.Network
14:38:52 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapError | 0xc0008e1000>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-dlct2d" to finish creating: GET https://rp.j9095680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d",
      "name": "6d3ea23e-abd5-4169-a804-e84509b8668d",
      "status": "Failed",
      "startTime": "2026-03-18T15:37:05.133933797Z",
      "endTime": "2026-03-18T15:38:56.831751444Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-dlct2d/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-dlct2d\" to finish creating: GET https://rp.j9095680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d\",\n  \"name\": \"6d3ea23e-abd5-4169-a804-e84509b8668d\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:37:05.133933797Z\",\n  \"endTime\": \"2026-03-18T15:38:56.831751444Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-dlct2d/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008e0fe0>{
            msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-dlct2d\" to finish creating: GET https://rp.j9095680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d\",\n  \"name\": \"6d3ea23e-abd5-4169-a804-e84509b8668d\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:37:05.133933797Z\",\n  \"endTime\": \"2026-03-18T15:38:56.831751444Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-dlct2d/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again af...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapError | 0xc0008e1000>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-dlct2d" to finish creating: GET https://rp.j9095680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d",
      "name": "6d3ea23e-abd5-4169-a804-e84509b8668d",
      "status": "Failed",
      "startTime": "2026-03-18T15:37:05.133933797Z",
      "endTime": "2026-03-18T15:38:56.831751444Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-dlct2d/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-dlct2d\" to finish creating: GET https://rp.j9095680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d\",\n  \"name\": \"6d3ea23e-abd5-4169-a804-e84509b8668d\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:37:05.133933797Z\",\n  \"endTime\": \"2026-03-18T15:38:56.831751444Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-dlct2d/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008e0fe0>{
            msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-dlct2d\" to finish creating: GET https://rp.j9095680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6d3ea23e-abd5-4169-a804-e84509b8668d\",\n  \"name\": \"6d3ea23e-abd5-4169-a804-e84509b8668d\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:37:05.133933797Z\",\n  \"endTime\": \"2026-03-18T15:38:56.831751444Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get network security group 'customer-nsg-name' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-dlct2d/providers/Microsoft.Network/networkSecurityGroups/customer-nsg-name\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again af...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head a2ca195 · merge 4da0d9f
 n/a provision failed ⚠#4508 (merged) 1
provision (1)
Error: failed to resolve sha256:<hex>: GET "<url>": invalid response; digest mismatch in Docker-Content-Digest: received "sha256:<hex>" when expecting "sha256:<hex>"
14:38:20 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.HypershiftOperator/global/mirror-image · suite step graph
Show raw failure
time=2026-03-18T15:15:08.938Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.HypershiftOperator resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> hypershift.image.registry <nil>}:{<nil> hypershift.image.repository <nil>}@{<nil> hypershift.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-18T15:15:13.751Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.HypershiftOperator resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image quay.io/redhat-services-prod/crt-redhat-acm-tenant/hypershift/hypershift-operator@sha256:7f027c55284ca274ff434b2866196b8392d0785bb97dc32e8d4abc18909fbdf7 to arohcpsvcdev.azurecr.io/redhat-services-prod/crt-redhat-acm-tenant/hypershift/hypershift-operator:7f027c55284ca274ff434b2866196b8392d0785bb97dc32e8d4abc18909fbdf7.\nThe image will still be available under it's original digest sha256:7f027c55284ca274ff434b2866196b8392d0785bb97dc32e8d4abc18909fbdf7 in the target registry.\nError: failed to resolve sha256:7f027c55284ca274ff434b2866196b8392d0785bb97dc32e8d4abc18909fbdf7: GET \"https://quay.io/v2/redhat-services-prod/crt-redhat-acm-tenant/hypershift/hypershift-operator/manifests/sha256:7f027c55284ca274ff434b2866196b8392d0785bb97dc32e8d4abc18909fbdf7\": invalid response; digest mismatch in Docker-Content-Digest: received \"sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\" when expecting \"sha256:7f027c55284ca274ff434b2866196b8392d0785bb97dc32e8d4abc18909fbdf7\"\n exit status 1"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a82c625 · merge a82c625
 n/a e2e failed #4347 (merged) 1
e2e (1)
ERROR CODE: MultipleErrorsOccurred; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift
14:37:49 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0006754c0>: 
    failed to create HCP cluster cluster-zstream-4-19-wjcpfb: failed waiting for cluster="cluster-zstream-4-19-wjcpfb" in resourcegroup="rg-zstream-upgrade-4-19-cct4x4" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120",
      "name": "161f28e8-f27b-44a6-985c-acbfac3cb120",
      "status": "Failed",
      "startTime": "2026-03-18T15:48:29.77050876Z",
      "endTime": "2026-03-18T15:49:46.080405554Z",
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "Cluster provisioning failed due to multiple errors",
        "details": [
          {
            "code": "InternalServerError",
            "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
          },
          {
            "code": "InternalServerError",
            "message": "Failed to get virtual network 'customer-vnet-zstream-wjcpfb' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cct4x4/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-wjcpfb\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-wjcpfb: failed waiting for cluster=\"cluster-zstream-4-19-wjcpfb\" in resourcegroup=\"rg-zstream-upgrade-4-19-cct4x4\" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120\",\n  \"name\": \"161f28e8-f27b-44a6-985c-acbfac3cb120\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:48:29.77050876Z\",\n  \"endTime\": \"2026-03-18T15:49:46.080405554Z\",\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"Cluster provisioning failed due to multiple errors\",\n    \"details\": [\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"Failed to get virtual network 'customer-vnet-zstream-wjcpfb' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cct4x4/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-wjcpfb\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0006754a0>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-19-wjcpfb\" in resourcegroup=\"rg-zstream-upgrade-4-19-cct4x4\" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfa...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0006754c0>: 
    failed to create HCP cluster cluster-zstream-4-19-wjcpfb: failed waiting for cluster="cluster-zstream-4-19-wjcpfb" in resourcegroup="rg-zstream-upgrade-4-19-cct4x4" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120",
      "name": "161f28e8-f27b-44a6-985c-acbfac3cb120",
      "status": "Failed",
      "startTime": "2026-03-18T15:48:29.77050876Z",
      "endTime": "2026-03-18T15:49:46.080405554Z",
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "Cluster provisioning failed due to multiple errors",
        "details": [
          {
            "code": "InternalServerError",
            "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
          },
          {
            "code": "InternalServerError",
            "message": "Failed to get virtual network 'customer-vnet-zstream-wjcpfb' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cct4x4/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-wjcpfb\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-wjcpfb: failed waiting for cluster=\"cluster-zstream-4-19-wjcpfb\" in resourcegroup=\"rg-zstream-upgrade-4-19-cct4x4\" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfac3cb120\",\n  \"name\": \"161f28e8-f27b-44a6-985c-acbfac3cb120\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:48:29.77050876Z\",\n  \"endTime\": \"2026-03-18T15:49:46.080405554Z\",\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"Cluster provisioning failed due to multiple errors\",\n    \"details\": [\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-243/providers/Microsoft.ManagedIdentity/userAssignedIdentities/disk-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"Failed to get virtual network 'customer-vnet-zstream-wjcpfb' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-19-cct4x4/providers/Microsoft.Network/virtualNetworks/customer-vnet-zstream-wjcpfb\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0006754a0>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-19-wjcpfb\" in resourcegroup=\"rg-zstream-upgrade-4-19-cct4x4\" to finish creating: GET https://rp.j2239104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/161f28e8-f27b-44a6-985c-acbfa...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 9baa329 · merge 9baa329
 n/a e2e failed #4461 (merged) 1
e2e (1)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity
14:28:09 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapError | 0xc00136e440>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sbghmb" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c",
      "name": "7163cf65-26e8-4c55-8654-25f332c3301c",
      "status": "Failed",
      "startTime": "2026-03-18T15:39:30.642744115Z",
      "endTime": "2026-03-18T15:40:26.468431346Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sbghmb\" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\",\n  \"name\": \"7163cf65-26e8-4c55-8654-25f332c3301c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:39:30.642744115Z\",\n  \"endTime\": \"2026-03-18T15:40:26.468431346Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00136e420>{
            msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sbghmb\" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\",\n  \"name\": \"7163cf65-26e8-4c55-8654-25f332c3301c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:39:30.642744115Z\",\n  \"endTime\": \"2026-03-18T15:40:26.468431346Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapError | 0xc00136e440>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sbghmb" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c",
      "name": "7163cf65-26e8-4c55-8654-25f332c3301c",
      "status": "Failed",
      "startTime": "2026-03-18T15:39:30.642744115Z",
      "endTime": "2026-03-18T15:40:26.468431346Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sbghmb\" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\",\n  \"name\": \"7163cf65-26e8-4c55-8654-25f332c3301c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:39:30.642744115Z\",\n  \"endTime\": \"2026-03-18T15:40:26.468431346Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00136e420>{
            msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-sbghmb\" to finish creating: GET https://rp.j8876800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7163cf65-26e8-4c55-8654-25f332c3301c\",\n  \"name\": \"7163cf65-26e8-4c55-8654-25f332c3301c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:39:30.642744115Z\",\n  \"endTime\": \"2026-03-18T15:40:26.468431346Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-151/providers/Microsoft.ManagedIdentity/userAssignedIdentities/file-csi-driver\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n-------------------------------------------------...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 4955535
 n/a e2e failed #4496 (open) 2
e2e (2)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity
14:25:38 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:135]: Unexpected error:
    <*exported.ResponseError | 0xc0006c71d0>: 
    GET https://rp.j1418880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c6784506-194c-4bbc-ad90-7a9253307720
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c6784506-194c-4bbc-ad90-7a9253307720",
      "name": "c6784506-194c-4bbc-ad90-7a9253307720",
      "status": "Failed",
      "startTime": "2026-03-18T15:42:08.505361166Z",
      "endTime": "2026-03-18T15:43:16.011043158Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InternalServerError",
        StatusCode: 200,
        RawResponse: {
            Status: "200 OK",
            StatusCode: 200,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Length": ["1581"],
                "X-Envoy-Upstream-Service-Time": ["9"],
                "Server": ["istio-envoy"],
                "Content-Type": ["application/json"],
                "X-Ms-Request-Id": [
                    "7b736841-ba39-45f8-96c3-140caef5a721",
                ],
                "Date": [
                    "Wed, 18 Mar 2026 15:50:42 GMT",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000aa2660>{
                s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 99, 54, 55, 56, 52, 53, 48, 54, 45, 49, 57, 52, 99, 45, 52, 98, 98, 99, 45, 97, 100, 57, 48, 45, 55, 97, 57, 50, 53, 51, 51, 48, 55, 55, 50, 48, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 99, 54, 55, 56, 52, 53, 48, 54, 45, 49, 57, 52, 99, 45, 52, 98, 98, 99, 45, 97, 100, 57, 48, 45, 55, 97, 57, 50, 53, 51, 51, 48, 55, 55, 50, 48, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 52, 50, 58, 48, 56, 46, 53, 48, 53, 51, 54, 49, 49, 54, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 52, 51, 58, 49, 54, 46, 48, 49, 49, 48, 52, 51, 49, 53, 56, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 54, 55, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 99, 108, 111, 117, 100, 45, 99, 111, 110, 116, 114, 111, 108, 108, 101, 114, 45, 109, 97, 110, 97, 103, 101, 114, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:135]: Unexpected error:
    <*exported.ResponseError | 0xc0006c71d0>: 
    GET https://rp.j1418880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c6784506-194c-4bbc-ad90-7a9253307720
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c6784506-194c-4bbc-ad90-7a9253307720",
      "name": "c6784506-194c-4bbc-ad90-7a9253307720",
      "status": "Failed",
      "startTime": "2026-03-18T15:42:08.505361166Z",
      "endTime": "2026-03-18T15:43:16.011043158Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-67/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InternalServerError",
        StatusCode: 200,
        RawResponse: {
            Status: "200 OK",
            StatusCode: 200,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Length": ["1581"],
                "X-Envoy-Upstream-Service-Time": ["9"],
                "Server": ["istio-envoy"],
                "Content-Type": ["application/json"],
                "X-Ms-Request-Id": [
                    "7b736841-ba39-45f8-96c3-140caef5a721",
                ],
                "Date": [
                    "Wed, 18 Mar 2026 15:50:42 GMT",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000aa2660>{
                s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 99, 54, 55, 56, 52, 53, 48, 54, 45, 49, 57, 52, 99, 45, 52, 98, 98, 99, 45, 97, 100, 57, 48, 45, 55, 97, 57, 50, 53, 51, 51, 48, 55, 55, 50, 48, 34, 44, 10, 32, 32, 32, 32, 34, 110, 97, 109, 101, 34, 58, 32, 34, 99, 54, 55, 56, 52, 53, 48, 54, 45, 49, 57, 52, 99, 45, 52, 98, 98, 99, 45, 97, 100, 57, 48, 45, 55, 97, 57, 50, 53, 51, 51, 48, 55, 55, 50, 48, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 116, 117, 115, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 34, 44, 10, 32, 32, 32, 32, 34, 115, 116, 97, 114, 116, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 52, 50, 58, 48, 56, 46, 53, 48, 53, 51, 54, 49, 49, 54, 54, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 110, 100, 84, 105, 109, 101, 34, 58, 32, 34, 50, 48, 50, 54, 45, 48, 51, 45, 49, 56, 84, 49, 53, 58, 52, 51, 58, 49, 54, 46, 48, 49, 49, 48, 52, 51, 49, 53, 56, 90, 34, 44, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 102, 97, 105, 108, 101, 100, 32, 116, 111, 32, 103, 101, 116, 32, 109, 97, 110, 97, 103, 101, 100, 32, 105, 100, 101, 110, 116, 105, 116, 121, 32, 39, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 114, 101, 115, 111, 117, 114, 99, 101, 71, 114, 111, 117, 112, 115, 47, 97, 114, 111, 45, 104, 99, 112, 45, 116, 101, 115, 116, 45, 109, 115, 105, 45, 99, 111, 110, 116, 97, 105, 110, 101, 114, 115, 45, 100, 101, 118, 45, 54, 55, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 77, 97, 110, 97, 103, 101, 100, 73, 100, 101, 110, 116, 105, 116, 121, 47, 117, 115, 101, 114, 65, 115, 115, 105, 103, 110, 101, 100, 73, 100, 101, 110, 116, 105, 116, 105, 101, 115, 47, 99, 108, 111, 117, 100, 45, 99, 111, 110, 116, 114, 111, 108, 108, 101, 114, 45, 109, 97, 110, 97, 103, 101, 114, 39, 58, 32, 71, 69, 84, 32, 104, 116, 116, 112, 115, 58, 47, 47, 109, 97, 110, 97, 103, 101, 109, 101, 110, 116, 46, 97, 122, 117, 114, 101, 46, 99, 111, 109, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity
14:25:38 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000b046a0>: 
    failed to create HCP cluster cluster-zstream-4-19-7bt5td: failed waiting for cluster="cluster-zstream-4-19-7bt5td" in resourcegroup="rg-zstream-upgrade-4-19-b9prxw" to finish creating: GET https://rp.j1418880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8",
      "name": "96c91453-1ae7-4e80-992f-9ac2bf0c1af8",
      "status": "Failed",
      "startTime": "2026-03-18T15:40:25.336190273Z",
      "endTime": "2026-03-18T15:41:06.719672779Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-7bt5td: failed waiting for cluster=\"cluster-zstream-4-19-7bt5td\" in resourcegroup=\"rg-zstream-upgrade-4-19-b9prxw\" to finish creating: GET https://rp.j1418880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8\",\n  \"name\": \"96c91453-1ae7-4e80-992f-9ac2bf0c1af8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:40:25.336190273Z\",\n  \"endTime\": \"2026-03-18T15:41:06.719672779Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000b04680>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-19-7bt5td\" in resourcegroup=\"rg-zstream-upgrade-4-19-b9prxw\" to finish creating: GET https://rp.j1418880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8\",\n  \"name\": \"96c91453-1ae7-4e80-992f-9ac2bf0c1af8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:40:25.336190273Z\",\n  \"endTime\": \"2026-03-18T15:41:06.719672779Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000b046a0>: 
    failed to create HCP cluster cluster-zstream-4-19-7bt5td: failed waiting for cluster="cluster-zstream-4-19-7bt5td" in resourcegroup="rg-zstream-upgrade-4-19-b9prxw" to finish creating: GET https://rp.j1418880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8",
      "name": "96c91453-1ae7-4e80-992f-9ac2bf0c1af8",
      "status": "Failed",
      "startTime": "2026-03-18T15:40:25.336190273Z",
      "endTime": "2026-03-18T15:41:06.719672779Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-7bt5td: failed waiting for cluster=\"cluster-zstream-4-19-7bt5td\" in resourcegroup=\"rg-zstream-upgrade-4-19-b9prxw\" to finish creating: GET https://rp.j1418880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8\",\n  \"name\": \"96c91453-1ae7-4e80-992f-9ac2bf0c1af8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:40:25.336190273Z\",\n  \"endTime\": \"2026-03-18T15:41:06.719672779Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000b04680>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-19-7bt5td\" in resourcegroup=\"rg-zstream-upgrade-4-19-b9prxw\" to finish creating: GET https://rp.j1418880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/96c91453-1ae7-4e80-992f-9ac2bf0c1af8\",\n  \"name\": \"96c91453-1ae7-4e80-992f-9ac2bf0c1af8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T15:40:25.336190273Z\",\n  \"endTime\": \"2026-03-18T15:41:06.719672779Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-2/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 7b93e84 · merge a0ece52
 n/a n/a passed #4481 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 16b44df · merge 3a9b5c9
 n/a e2e failed #4503 (merged) 1
e2e (1)
ERROR CODE: InternalServerError; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.ManagedIdentity
12:37:55 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error:
    <*fmt.wrapError | 0xc000d40360>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-nwdhx9" to finish creating: GET https://rp.j4226432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3",
      "name": "4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3",
      "status": "Failed",
      "startTime": "2026-03-18T13:27:52.504509033Z",
      "endTime": "2026-03-18T13:29:26.947856695Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-nwdhx9\" to finish creating: GET https://rp.j4226432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\",\n  \"name\": \"4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T13:27:52.504509033Z\",\n  \"endTime\": \"2026-03-18T13:29:26.947856695Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000d40340>{
            msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-nwdhx9\" to finish creating: GET https://rp.j4226432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\",\n  \"name\": \"4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T13:27:52.504509033Z\",\n  \"endTime\": \"2026-03-18T13:29:26.947856695Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error:
    <*fmt.wrapError | 0xc000d40360>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-nwdhx9" to finish creating: GET https://rp.j4226432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3",
      "name": "4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3",
      "status": "Failed",
      "startTime": "2026-03-18T13:27:52.504509033Z",
      "endTime": "2026-03-18T13:29:26.947856695Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-nwdhx9\" to finish creating: GET https://rp.j4226432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\",\n  \"name\": \"4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T13:27:52.504509033Z\",\n  \"endTime\": \"2026-03-18T13:29:26.947856695Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"SubscriptionRequestsThrottled\\\",\\n    \\\"message\\\": \\\"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'd6b62dfa-87f5-49b3-bbcb-4a687c4faa96' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000d40340>{
            msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-nwdhx9\" to finish creating: GET https://rp.j4226432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\",\n  \"name\": \"4e2f64df-6d2a-4cdf-bddb-1c2e60f9a6f3\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T13:27:52.504509033Z\",\n  \"endTime\": \"2026-03-18T13:29:26.947856695Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager': GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-91/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\\n--------------------------------------------------------------------------------\\nRESPONSE 429: 429 Too Many Requests\\nERROR CODE: SubscriptionRequestsThrottled\\n---------------...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 476f04d
 n/a unknown failed #4403 (open) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 30983c7 · merge 5a75b4f
 n/a n/a passed #4491 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5a75b4f · merge 5a75b4f
 n/a n/a passed #4491 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 30983c7 · merge 5a75b4f
 n/a e2e failed #4491 (merged) 3
e2e (3)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:26:28 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000e023d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-2v58jh" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000eb1140>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-2v58jh\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000eb1120>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-2v58jh\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000eb1100>{
                        msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-2v58jh\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000e023d8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-2v58jh" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000eb1140>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-2v58jh\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000eb1120>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-2v58jh\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000eb1100>{
                        msg: "failed waiting for hcpCluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-2v58jh\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:26:28 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001584240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-list4k2n65" in resourcegroup="ea-list-vdf4nx" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0004810c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list4k2n65\" in resourcegroup=\"ea-list-vdf4nx\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0004810a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list4k2n65\" in resourcegroup=\"ea-list-vdf4nx\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000481080>{
                        msg: "failed waiting for hcpCluster=\"ea-list4k2n65\" in resourcegroup=\"ea-list-vdf4nx\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001584240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-list4k2n65" in resourcegroup="ea-list-vdf4nx" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0004810c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list4k2n65\" in resourcegroup=\"ea-list-vdf4nx\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0004810a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ea-list4k2n65\" in resourcegroup=\"ea-list-vdf4nx\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000481080>{
                        msg: "failed waiting for hcpCluster=\"ea-list4k2n65\" in resourcegroup=\"ea-list-vdf4nx\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
12:26:28 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000c82c30>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-5x6tvh" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009494c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-5x6tvh\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009494a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-5x6tvh\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000949480>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-5x6tvh\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000c82c30>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-5x6tvh" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc0009494c0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-5x6tvh\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc0009494a0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-5x6tvh\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000949480>{
                        msg: "failed waiting for hcpCluster=\"basic-hcp-cluster\" in resourcegroup=\"customer-rg-5x6tvh\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb09b0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 898324b
 n/a unknown failed ★#4192 (open) 1
unknown (1)
Process did not finish before 2h0m0s timeout
12:24:18 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit
Show raw failure
{"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-03-18T15:11:59Z"}
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5caac0b
 n/a unknown failed #4193 (open) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5fab4cf
 n/a unknown failed #4193 (open) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head b3b9232 · merge b6d9c64
 n/a unknown failed #4494 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head b6f3e3f · merge 9feee09
 n/a provision failed #4004 (merged) 1
provision (1)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError("HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve
11:23:53 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RP.Backend/global/mirror-image · suite step graph
Show raw failure
time=2026-03-18T12:03:36.219Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> backend.image.registry <nil>}:{<nil> backend.image.repository <nil>}@{<nil> backend.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-18T12:03:48.613Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nSetting up registry authentication for CI source registry.\ninfo: Using registry public hostname registry.build02.ci.openshift.org\nSaved credentials for registry.build02.ci.openshift.org into /tmp/tmp.3jYMm47kT5/containers/auth.json\nLogging into target ACR arohcpsvcdev.\nFailed to log in to ACR arohcpsvcdev: ERROR: The command failed with an unexpected error. Here is the traceback:\nERROR: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 204, in _new_conn\n    sock = connection.create_connection(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/connection.py\", line 60, in create_connection\n    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/python3.12/socket.py\", line 963, in getaddrinfo\n    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 787, in urlopen\n    response = self._make_request(\n               ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 488, in _make_request\n    raise new_e\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 464, in _make_request\n    self._validate_conn(conn)\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 1093, in _validate_conn\n    conn.connect()\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 759, in connect\n    self.sock = sock = self._new_conn()\n                       ^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 211, in _new_conn\n    raise NameResolutionError(self.host, self, e) from e\nurllib3.exceptions.NameResolutionError: HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n    resp = conn.urlopen(\n           ^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 841, in urlopen\n    retries = retries.increment(\n              ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/retry.py\", line 535, in increment\n    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/knack/cli.py\", line 233, in invoke\n    cmd_result = self.invocation.execute(args)\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 682, in execute\n    raise ex\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 812, in _run_jobs_serially\n    results.append(self._run_job(expanded_arg, cmd_copy))\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 781, in _run_job\n    result = cmd_copy(params)\n             ^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 336, in __call__\n    return self.handler(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py\", line 120, in handler\n    return op(**command_args)\n           ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/custom.py\", line 294, in acr_login\n    login_server, _, password = get_login_credentials(\n                                ^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 505, in get_login_credentials\n    return _get_credentials(cmd,\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 449, in _get_credentials\n    return login_server, EMPTY_GUID, _get_aad_token(cli_ctx,\n                                     ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 285, in _get_aad_token\n    return _get_aad_token_after_challenge(cli_ctx,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 159, in _get_aad_token_after_challenge\n    response = requests.post(url=authhost, data=urlencode(content), headers=headers,\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 115, in post\n    return request(\"post\", url, data=data, json=json, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n    return session.request(method=method, url=url, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 700, in send\n    raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTo check existing issues, please visit: https://github.com/Azure/azure-cli/issues\n exit status 1"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 131806c · merge 131806c
 n/a n/a passed #4351 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 8597c39
 n/a n/a passed #4504 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head a088e35
 n/a n/a passed #4281 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 16b44df · merge 3a9b5c9
 n/a n/a passed #4503 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head ad7003e · merge ad7003e
 n/a n/a passed #4188 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 02baee7
 n/a provision failed #4403 (open) 1
provision (1)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError("HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve
10:04:57 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Maestro.Agent/global/mirror-image · suite step graph
Show raw failure
time=2026-03-18T10:38:19.959Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> maestro.image.registry <nil>}:{<nil> maestro.image.repository <nil>}@{<nil> maestro.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-18T10:38:33.511Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Agent resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nFailed to log in to ACR arohcpsvcdev: ERROR: The command failed with an unexpected error. Here is the traceback:\nERROR: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 204, in _new_conn\n    sock = connection.create_connection(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/connection.py\", line 60, in create_connection\n    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/python3.12/socket.py\", line 963, in getaddrinfo\n    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 787, in urlopen\n    response = self._make_request(\n               ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 488, in _make_request\n    raise new_e\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 464, in _make_request\n    self._validate_conn(conn)\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 1093, in _validate_conn\n    conn.connect()\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 759, in connect\n    self.sock = sock = self._new_conn()\n                       ^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 211, in _new_conn\n    raise NameResolutionError(self.host, self, e) from e\nurllib3.exceptions.NameResolutionError: HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n    resp = conn.urlopen(\n           ^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 841, in urlopen\n    retries = retries.increment(\n              ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/retry.py\", line 535, in increment\n    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/knack/cli.py\", line 233, in invoke\n    cmd_result = self.invocation.execute(args)\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 682, in execute\n    raise ex\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 812, in _run_jobs_serially\n    results.append(self._run_job(expanded_arg, cmd_copy))\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 781, in _run_job\n    result = cmd_copy(params)\n             ^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 336, in __call__\n    return self.handler(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py\", line 120, in handler\n    return op(**command_args)\n           ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/custom.py\", line 294, in acr_login\n    login_server, _, password = get_login_credentials(\n                                ^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 505, in get_login_credentials\n    return _get_credentials(cmd,\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 449, in _get_credentials\n    return login_server, EMPTY_GUID, _get_aad_token(cli_ctx,\n                                     ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 276, in _get_aad_token\n    token_params = _handle_challenge_phase(\n                   ^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 100, in _handle_challenge_phase\n    challenge = requests.get(request_url, verify=not should_disable_connection_verify())\n                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 73, in get\n    return request(\"get\", url, params=params, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n    return session.request(method=method, url=url, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 700, in send\n    raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /v2/ (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTo check existing issues, please visit: https://github.com/Azure/azure-cli/issues\n exit status 1"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 3d35775 · merge 3d35775
 n/a n/a passed #4502 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 30983c7 · merge 5a75b4f
 n/a n/a passed #4491 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 299f5e8 · merge a0ece52
 n/a unknown failed #4481 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0d2c310 · merge 0d2c310
 n/a n/a passed #4224 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head d44789f
 n/a n/a passed #4403 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 30983c7 · merge 5a75b4f
 n/a n/a passed #4491 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c818402 · merge 8550ece
 n/a unknown failed #4219 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ad7003e · merge ad7003e
 n/a n/a passed #4188 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 30c99a5 · merge 30c99a5
 n/a n/a passed #4498 (merged) 0 n/a