CIHealth
<
Single Day: Mar 5▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head c23c6e9
 n/a n/a passed #3973 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 77248e3 · merge 77248e3
 n/a n/a passed #4334 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 3f1edac · merge 77248e3
 n/a unknown failed #4334 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 6f76269
 n/a unknown failed #3973 (open) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 3e32601 · merge 72fc910
 n/a n/a passed #4340 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head fc55efa · merge 8550ece
 n/a n/a passed #4219 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 1a9edf7 · merge 1a9edf7
 n/a n/a passed #4339 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 7cbe20c · merge 7cbe20c
 n/a n/a passed #4218 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 7cbe20c · merge 7cbe20c
 n/a n/a passed #4218 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7fbd7a3 · merge 7fbd7a3
 n/a n/a passed #4266 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9658e7f
 n/a n/a passed #4330 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 85881a9 · merge 19c0b46
 n/a n/a passed #4338 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head bad9200 · merge 8550ece
 n/a n/a passed #4219 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 396bcea
 n/a e2e failed ⚠#4333 (closed) 5
e2e (5)
ERROR CODE: InvalidRequestContent; detail message Forbidden: field is immutable; provider Microsoft.RedHatOpenShift
18:31:45 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:415]: Unexpected error:
    <*exported.ResponseError | 0xc000e56d20>: 
    PUT https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-7gwkwc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Forbidden: field is immutable",
        "target": "trackedResource.resource.systemData.createdAt"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Request-Id": [
                    "6656c588-1172-4de4-835b-a012ecffe1b0",
                ],
                "Date": [
                    "Thu, 05 Mar 2026 19:55:31 GMT",
                ],
                "Content-Length": ["183"],
                "X-Envoy-Upstream-Service-Time": ["76"],
                "Server": ["istio-envoy"],
                "Content-Type": ["application/json"],
            },
            Body: <*exported.nopClosingBytesReader | 0xc00039db80>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 116, 114, 97, 99, 107, 101, 100, 82, 101, 115, 111, 117, 114, 99, 101, 46, 114, 101, 115, 111, 117, 114, 99, 101, 46, 115, 121, 115, 116, 101, 109, 68, 97, 116, 97, 46, 99, 114, 101, 97, 116, 101, 100, 65, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 183,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PUT",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "rp.j7796480.hcpsvc.osadev.cloud",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-7gwkwc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2024-06-10-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Content-Length": ["7478"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
                    ],
                    "X-Ms-Arm-Resource-System-Data": [
                        "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-05T19:55:31Z\"}",
                    ],
                    "X-Ms-Identity-Url": [
                        "https://dummyhost.identity.azure.net",
                    ],
                    "Authorization": ["redacted"],
                    "Accept": ["application/json"],
                },
                Body: <*http.readTrackingBody | 0xc000e76c78>{
                    ReadCloser: <*runtime.retryableRequestBody | 0xc0000983d0>{
                        body: <exported.nopCloser>{
                            ReadSeeker: <*bytes.Reader | 0xc000e569c0>{
                                s: "{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-7gwkwc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test\",\"identity\":{\"type\":\"UserAssigned\",\"userAssignedIdentities\":{\"...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:415]: Unexpected error:
    <*exported.ResponseError | 0xc000e56d20>: 
    PUT https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-7gwkwc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Forbidden: field is immutable",
        "target": "trackedResource.resource.systemData.createdAt"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Request-Id": [
                    "6656c588-1172-4de4-835b-a012ecffe1b0",
                ],
                "Date": [
                    "Thu, 05 Mar 2026 19:55:31 GMT",
                ],
                "Content-Length": ["183"],
                "X-Envoy-Upstream-Service-Time": ["76"],
                "Server": ["istio-envoy"],
                "Content-Type": ["application/json"],
            },
            Body: <*exported.nopClosingBytesReader | 0xc00039db80>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 116, 114, 97, 99, 107, 101, 100, 82, 101, 115, 111, 117, 114, 99, 101, 46, 114, 101, 115, 111, 117, 114, 99, 101, 46, 115, 121, 115, 116, 101, 109, 68, 97, 116, 97, 46, 99, 114, 101, 97, 116, 101, 100, 65, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 183,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PUT",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "rp.j7796480.hcpsvc.osadev.cloud",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-7gwkwc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2024-06-10-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Content-Length": ["7478"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
                    ],
                    "X-Ms-Arm-Resource-System-Data": [
                        "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-05T19:55:31Z\"}",
                    ],
                    "X-Ms-Identity-Url": [
                        "https://dummyhost.identity.azure.net",
                    ],
                    "Authorization": ["redacted"],
                    "Accept": ["application/json"],
                },
                Body: <*http.readTrackingBody | 0xc000e76c78>{
                    ReadCloser: <*runtime.retryableRequestBody | 0xc0000983d0>{
                        body: <exported.nopCloser>{
                            ReadSeeker: <*bytes.Reader | 0xc000e569c0>{
                                s: "{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-7gwkwc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test\",\"identity\":{\"type\":\"UserAssigned\",\"userAssignedIdentities\":{\"...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: InvalidRequestContent; detail message Forbidden: field is immutable; provider Microsoft.ManagedIdentity
18:31:45 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:192]: Unexpected error:
    <*exported.ResponseError | 0xc000830ab0>: 
    PATCH https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-x5j9d5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Forbidden: field is immutable",
        "target": "trackedResource.resource.systemData.createdAt"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Request-Id": [
                    "ff37132a-131c-4148-9434-beb55eb4e61b",
                ],
                "Date": [
                    "Thu, 05 Mar 2026 19:40:05 GMT",
                ],
                "Content-Length": ["183"],
                "X-Envoy-Upstream-Service-Time": ["67"],
                "Server": ["istio-envoy"],
            },
            Body: <*exported.nopClosingBytesReader | 0xc0009be360>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 116, 114, 97, 99, 107, 101, 100, 82, 101, 115, 111, 117, 114, 99, 101, 46, 114, 101, 115, 111, 117, 114, 99, 101, 46, 115, 121, 115, 116, 101, 109, 68, 97, 116, 97, 46, 99, 114, 101, 97, 116, 101, 100, 65, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 183,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "rp.j7796480.hcpsvc.osadev.cloud",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-x5j9d5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2024-06-10-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Authorization": ["redacted"],
                    "Content-Length": ["1911"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
                    ],
                    "X-Ms-Arm-Resource-System-Data": [
                        "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-05T19:40:05Z\"}",
                    ],
                    "X-Ms-Identity-Url": [
                        "https://dummyhost.identity.azure.net",
                    ],
                    "Accept": ["application/json"],
                },
                Body: <*http.readTrackingBody | 0xc001109668>{
                    ReadCloser: <*runtime.retryableRequestBody | 0xc00125f1d0>{
                        body: <exported.nopCloser>{
                            ReadSeeker: <*bytes.Reader | 0xc0008c88a0>{
                                s: "{\"identity\":{\"type\":\"UserAssigned\",\"userAssignedIdentities\":{\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-184/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\":{},\"/subsc...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:192]: Unexpected error:
    <*exported.ResponseError | 0xc000830ab0>: 
    PATCH https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-x5j9d5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Forbidden: field is immutable",
        "target": "trackedResource.resource.systemData.createdAt"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Request-Id": [
                    "ff37132a-131c-4148-9434-beb55eb4e61b",
                ],
                "Date": [
                    "Thu, 05 Mar 2026 19:40:05 GMT",
                ],
                "Content-Length": ["183"],
                "X-Envoy-Upstream-Service-Time": ["67"],
                "Server": ["istio-envoy"],
            },
            Body: <*exported.nopClosingBytesReader | 0xc0009be360>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 116, 114, 97, 99, 107, 101, 100, 82, 101, 115, 111, 117, 114, 99, 101, 46, 114, 101, 115, 111, 117, 114, 99, 101, 46, 115, 121, 115, 116, 101, 109, 68, 97, 116, 97, 46, 99, 114, 101, 97, 116, 101, 100, 65, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 183,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "rp.j7796480.hcpsvc.osadev.cloud",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-x5j9d5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2024-06-10-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Authorization": ["redacted"],
                    "Content-Length": ["1911"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
                    ],
                    "X-Ms-Arm-Resource-System-Data": [
                        "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-05T19:40:05Z\"}",
                    ],
                    "X-Ms-Identity-Url": [
                        "https://dummyhost.identity.azure.net",
                    ],
                    "Accept": ["application/json"],
                },
                Body: <*http.readTrackingBody | 0xc001109668>{
                    ReadCloser: <*runtime.retryableRequestBody | 0xc00125f1d0>{
                        body: <exported.nopCloser>{
                            ReadSeeker: <*bytes.Reader | 0xc0008c88a0>{
                                s: "{\"identity\":{\"type\":\"UserAssigned\",\"userAssignedIdentities\":{\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-dev-184/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\":{},\"/subsc...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: InvalidRequestContent; detail message Forbidden: field is immutable; provider Microsoft.RedHatOpenShift
18:31:45 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:152]: Unexpected error:
    <*exported.ResponseError | 0xc000ce4330>: 
    PATCH https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-t8rzhg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-update-nodes
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Forbidden: field is immutable",
        "target": "trackedResource.resource.systemData.createdAt"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Date": [
                    "Thu, 05 Mar 2026 20:05:02 GMT",
                ],
                "Content-Length": ["183"],
                "X-Envoy-Upstream-Service-Time": ["103"],
                "Server": ["istio-envoy"],
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Request-Id": [
                    "e4540555-2b10-4e16-9fbb-b3ca8af89c78",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000f39700>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 116, 114, 97, 99, 107, 101, 100, 82, 101, 115, 111, 117, 114, 99, 101, 46, 114, 101, 115, 111, 117, 114, 99, 101, 46, 115, 121, 115, 116, 101, 109, 68, 97, 116, 97, 46, 99, 114, 101, 97, 116, 101, 100, 65, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 183,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "rp.j7796480.hcpsvc.osadev.cloud",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-t8rzhg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-update-nodes",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2024-06-10-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Authorization": ["redacted"],
                    "X-Ms-Arm-Resource-System-Data": [
                        "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-05T20:05:01Z\"}",
                    ],
                    "X-Ms-Identity-Url": [
                        "https://dummyhost.identity.azure.net",
                    ],
                    "Accept": ["application/json"],
                    "Content-Length": ["29"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
                    ],
                },
                Body: <*http.readTrackingBody | 0xc000f915d8>{
                    ReadCloser: <*runtime.retryableRequestBody | 0xc0008af0e0>{
                        body: <exported.nopCloser>{
                            ReadSeeker: <*bytes.Reader | 0xc001529d10>{
                                s: "{\"properties\":{\"replicas\":3}}",
                                i: 29,
                                prevRune: -1,
                            },
                        },
                    },
                    didR...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:152]: Unexpected error:
    <*exported.ResponseError | 0xc000ce4330>: 
    PATCH https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-t8rzhg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-update-nodes
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Forbidden: field is immutable",
        "target": "trackedResource.resource.systemData.createdAt"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Date": [
                    "Thu, 05 Mar 2026 20:05:02 GMT",
                ],
                "Content-Length": ["183"],
                "X-Envoy-Upstream-Service-Time": ["103"],
                "Server": ["istio-envoy"],
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Request-Id": [
                    "e4540555-2b10-4e16-9fbb-b3ca8af89c78",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000f39700>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 116, 114, 97, 99, 107, 101, 100, 82, 101, 115, 111, 117, 114, 99, 101, 46, 114, 101, 115, 111, 117, 114, 99, 101, 46, 115, 121, 115, 116, 101, 109, 68, 97, 116, 97, 46, 99, 114, 101, 97, 116, 101, 100, 65, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 183,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "rp.j7796480.hcpsvc.osadev.cloud",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-t8rzhg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-update-nodes-hcp-cluster/nodePools/np-update-nodes",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2024-06-10-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Authorization": ["redacted"],
                    "X-Ms-Arm-Resource-System-Data": [
                        "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-05T20:05:01Z\"}",
                    ],
                    "X-Ms-Identity-Url": [
                        "https://dummyhost.identity.azure.net",
                    ],
                    "Accept": ["application/json"],
                    "Content-Length": ["29"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
                    ],
                },
                Body: <*http.readTrackingBody | 0xc000f915d8>{
                    ReadCloser: <*runtime.retryableRequestBody | 0xc0008af0e0>{
                        body: <exported.nopCloser>{
                            ReadSeeker: <*bytes.Reader | 0xc001529d10>{
                                s: "{\"properties\":{\"replicas\":3}}",
                                i: 29,
                                prevRune: -1,
                            },
                        },
                    },
                    didR...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: InvalidRequestContent; detail message Forbidden: field is immutable; provider Microsoft.RedHatOpenShift
18:31:45 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:190]: Unexpected error:
    <*fmt.wrapError | 0xc000d0a580>: 
    failed creating external auth "ea-list" in resourcegroup="ea-list-7jrd2j" for cluster="ea-listmxjsn8": PUT https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-7jrd2j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmxjsn8/externalAuths/ea-list
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Forbidden: field is immutable",
        "target": "trackedResource.resource.systemData.createdAt"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-7jrd2j\" for cluster=\"ea-listmxjsn8\": PUT https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-7jrd2j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmxjsn8/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Forbidden: field is immutable\",\n    \"target\": \"trackedResource.resource.systemData.createdAt\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0009faed0>{
            ErrorCode: "InvalidRequestContent",
            StatusCode: 400,
            RawResponse: {
                Status: "400 Bad Request",
                StatusCode: 400,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": [
                        "InvalidRequestContent",
                    ],
                    "X-Ms-Request-Id": [
                        "a3660b5c-972d-46d2-a067-468fd2bc7b18",
                    ],
                    "Date": [
                        "Thu, 05 Mar 2026 19:40:18 GMT",
                    ],
                    "Content-Length": ["183"],
                    "X-Envoy-Upstream-Service-Time": ["43"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000d0a4c0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 116, 114, 97, 99, 107, 101, 100, 82, 101, 115, 111, 117, 114, 99, 101, 46, 114, 101, 115, 111, 117, 114, 99, 101, 46, 115, 121, 115, 116, 101, 109, 68, 97, 116, 97, 46, 99, 114, 101, 97, 116, 101, 100, 65, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 183,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "PUT",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j7796480.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-7jrd2j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmxjsn8/externalAuths/ea-list",
                        RawPath: "",
                        OmitHost: false,
                        ForceQuery: false,
                        RawQuery: "api-version=2024-06-10-preview",
                        Fragment: "",
                        RawFragment: "",
                    },
                    Proto: "HTTP/1.1",
                    ProtoMajor: 1,
                    ProtoMinor: 1,
                    Header: {
                        "X-Ms-Identity-Url": [
                            "https://dummyhost.identity.azure.net",
                        ],
                        "Ac...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:190]: Unexpected error:
    <*fmt.wrapError | 0xc000d0a580>: 
    failed creating external auth "ea-list" in resourcegroup="ea-list-7jrd2j" for cluster="ea-listmxjsn8": PUT https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-7jrd2j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmxjsn8/externalAuths/ea-list
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Forbidden: field is immutable",
        "target": "trackedResource.resource.systemData.createdAt"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-7jrd2j\" for cluster=\"ea-listmxjsn8\": PUT https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-7jrd2j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmxjsn8/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Forbidden: field is immutable\",\n    \"target\": \"trackedResource.resource.systemData.createdAt\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0009faed0>{
            ErrorCode: "InvalidRequestContent",
            StatusCode: 400,
            RawResponse: {
                Status: "400 Bad Request",
                StatusCode: 400,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Server": ["istio-envoy"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": [
                        "InvalidRequestContent",
                    ],
                    "X-Ms-Request-Id": [
                        "a3660b5c-972d-46d2-a067-468fd2bc7b18",
                    ],
                    "Date": [
                        "Thu, 05 Mar 2026 19:40:18 GMT",
                    ],
                    "Content-Length": ["183"],
                    "X-Envoy-Upstream-Service-Time": ["43"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000d0a4c0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 116, 114, 97, 99, 107, 101, 100, 82, 101, 115, 111, 117, 114, 99, 101, 46, 114, 101, 115, 111, 117, 114, 99, 101, 46, 115, 121, 115, 116, 101, 109, 68, 97, 116, 97, 46, 99, 114, 101, 97, 116, 101, 100, 65, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 183,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "PUT",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "rp.j7796480.hcpsvc.osadev.cloud",
                        Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-7jrd2j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmxjsn8/externalAuths/ea-list",
                        RawPath: "",
                        OmitHost: false,
                        ForceQuery: false,
                        RawQuery: "api-version=2024-06-10-preview",
                        Fragment: "",
                        RawFragment: "",
                    },
                    Proto: "HTTP/1.1",
                    ProtoMajor: 1,
                    ProtoMinor: 1,
                    Header: {
                        "X-Ms-Identity-Url": [
                            "https://dummyhost.identity.azure.net",
                        ],
                        "Ac...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
ERROR CODE: InvalidRequestContent; detail message Forbidden: field is immutable; provider Microsoft.RedHatOpenShift
18:31:45 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error:
    <*exported.ResponseError | 0xc00157aa20>: 
    PATCH https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-lggzhf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Forbidden: field is immutable",
        "target": "trackedResource.resource.systemData.createdAt"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Length": ["183"],
                "X-Envoy-Upstream-Service-Time": ["77"],
                "Server": ["istio-envoy"],
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Request-Id": [
                    "b4a90f0f-1b86-4d5b-8bd8-88595707a037",
                ],
                "Date": [
                    "Thu, 05 Mar 2026 19:49:00 GMT",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc00160e340>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 116, 114, 97, 99, 107, 101, 100, 82, 101, 115, 111, 117, 114, 99, 101, 46, 114, 101, 115, 111, 117, 114, 99, 101, 46, 115, 121, 115, 116, 101, 109, 68, 97, 116, 97, 46, 99, 114, 101, 97, 116, 101, 100, 65, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 183,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "rp.j7796480.hcpsvc.osadev.cloud",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-lggzhf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2024-06-10-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Content-Length": ["50"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
                    ],
                    "X-Ms-Arm-Resource-System-Data": [
                        "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-05T19:49:00Z\"}",
                    ],
                    "X-Ms-Identity-Url": [
                        "https://dummyhost.identity.azure.net",
                    ],
                    "Authorization": ["redacted"],
                    "Accept": ["application/json"],
                },
                Body: <*http.readTrackingBody | 0xc000b289c0>{
                    ReadCloser: <*runtime.retryableRequestBody | 0xc00117a470>{
                        body: <exported.nopCloser>{
                            ReadSeeker: <*bytes.Reader | 0xc0012b5e60>{
                                s: "{\"properties\":{\"autoscaling\":{\"maxNodesTotal\":3}}}",
                                i: 50,
                                prevRune: -1,
                            },
                        },
                    },
                    didRead: true,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error:
    <*exported.ResponseError | 0xc00157aa20>: 
    PATCH https://rp.j7796480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-lggzhf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Forbidden: field is immutable",
        "target": "trackedResource.resource.systemData.createdAt"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InvalidRequestContent",
        StatusCode: 400,
        RawResponse: {
            Status: "400 Bad Request",
            StatusCode: 400,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Content-Length": ["183"],
                "X-Envoy-Upstream-Service-Time": ["77"],
                "Server": ["istio-envoy"],
                "Content-Type": ["application/json"],
                "X-Ms-Error-Code": [
                    "InvalidRequestContent",
                ],
                "X-Ms-Request-Id": [
                    "b4a90f0f-1b86-4d5b-8bd8-88595707a037",
                ],
                "Date": [
                    "Thu, 05 Mar 2026 19:49:00 GMT",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc00160e340>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 118, 97, 108, 105, 100, 82, 101, 113, 117, 101, 115, 116, 67, 111, 110, 116, 101, 110, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 111, 114, 98, 105, 100, 100, 101, 110, 58, 32, 102, 105, 101, 108, 100, 32, 105, 115, 32, 105, 109, 109, 117, 116, 97, 98, 108, 101, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 116, 114, 97, 99, 107, 101, 100, 82, 101, 115, 111, 117, 114, 99, 101, 46, 114, 101, 115, 111, 117, 114, 99, 101, 46, 115, 121, 115, 116, 101, 109, 68, 97, 116, 97, 46, 99, 114, 101, 97, 116, 101, 100, 65, 116, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 183,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "rp.j7796480.hcpsvc.osadev.cloud",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-lggzhf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2024-06-10-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "Content-Length": ["50"],
                    "Content-Type": ["application/json"],
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7; linux)",
                    ],
                    "X-Ms-Arm-Resource-System-Data": [
                        "{\"createdBy\": \"e2e-test\", \"createdByType\": \"Application\", \"createdAt\": \"2026-03-05T19:49:00Z\"}",
                    ],
                    "X-Ms-Identity-Url": [
                        "https://dummyhost.identity.azure.net",
                    ],
                    "Authorization": ["redacted"],
                    "Accept": ["application/json"],
                },
                Body: <*http.readTrackingBody | 0xc000b289c0>{
                    ReadCloser: <*runtime.retryableRequestBody | 0xc00117a470>{
                        body: <exported.nopCloser>{
                            ReadSeeker: <*bytes.Reader | 0xc0012b5e60>{
                                s: "{\"properties\":{\"autoscaling\":{\"maxNodesTotal\":3}}}",
                                i: 50,
                                prevRune: -1,
                            },
                        },
                    },
                    didRead: true,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 0ac5e19 · merge 0ac5e19
 n/a n/a passed #4181 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head db95ca2 · merge db95ca2
 n/a provision failed #4321 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
18:19:58 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T18:59:54.847Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T18:59:56.517Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T18:59:57.630Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=829db07553358e04ba6f1974ce4382138f4ed683bf2846b0fd1f28bcc23e2098
time=2026-03-05T19:01:58.722Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j6003968/providers/Microsoft.Resources/deployments/829db07553358e04ba6f1974ce4382138f4ed683bf2846b0fd1f28bcc23e2098/operationStatuses/08584288696885440870\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6003968/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6003968/providers/Microsoft.Monitor/accounts/services-j6003968/providers/Microsoft.Authorization/roleAssignments/b92119a1-893c-5d44-8e94-8017dcd14434\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T190018Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'f8034c64-29df-4a64-a87d-1e6b3d8a13de', request correlation id 'a537b1c5-5960-4794-9504-fdb91cfb0c64'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head efd28c6 · merge efd28c6
 n/a provision failed #4284 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid
18:09:54 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T18:42:31.452Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T18:42:33.576Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T18:42:34.756Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=d3a2e71b310b48f0675124d7e66c16b8b1c647bdfde75727e3b66ae5d577428b
time=2026-03-05T18:44:05.514Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5108224/providers/Microsoft.Resources/deployments/d3a2e71b310b48f0675124d7e66c16b8b1c647bdfde75727e3b66ae5d577428b/operationStatuses/08584288707314078974\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5108224/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5108224/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j5108224\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 6815fb0b-9d2c-49ce-8487-a357f6c0aa55:3/5/2026 6:43:41 PM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 3f1edac · merge 77248e3
 n/a n/a passed #4334 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 70af027 · merge 70af027
 n/a n/a passed #4336 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 157355c · merge 40bc11d
 n/a e2e failed #4313 (merged) 16
e2e (16)
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc00122f140>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-kbvznc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-kbvznc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00110beb0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc",
            },
            <*fmt.wrapErrors | 0xc00122f110>{
                msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-kbvznc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00110beb0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc00122f140>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-kbvznc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-kbvznc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00110beb0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc",
            },
            <*fmt.wrapErrors | 0xc00122f110>{
                msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-kbvznc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00110beb0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-kbvznc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc0007ad080>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5nqvcb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-5nqvcb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00068fca0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb",
            },
            <*fmt.wrapErrors | 0xc0007ad020>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-5nqvcb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00068fca0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc0007ad080>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5nqvcb" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-5nqvcb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00068fca0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb",
            },
            <*fmt.wrapErrors | 0xc0007ad020>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-5nqvcb\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00068fca0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-5nqvcb",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error:
    <*fmt.wrapErrors | 0xc00133d5f0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp, error: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-hmsxvp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp, error: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-hmsxvp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00109fae0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp",
            },
            <*fmt.wrapErrors | 0xc00133d5c0>{
                msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-hmsxvp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00109fae0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:82]: Unexpected error:
    <*fmt.wrapErrors | 0xc00133d5f0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp, error: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-hmsxvp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster np-update-nodes-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp, error: failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-hmsxvp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00109fae0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp",
            },
            <*fmt.wrapErrors | 0xc00133d5c0>{
                msg: "failed waiting for cluster=\"np-update-nodes-hcp-cluster\" in resourcegroup=\"nodepool-update-nodes-hmsxvp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00109fae0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-hmsxvp",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error:
    <*fmt.wrapErrors | 0xc0016e51a0>: 
    failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-vcq5ns" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns, error: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-vcq5ns\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0016c03a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns",
            },
            <*fmt.wrapErrors | 0xc0016e5170>{
                msg: "failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-vcq5ns\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0016c03a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:106]: Unexpected error:
    <*fmt.wrapErrors | 0xc0016e51a0>: 
    failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-vcq5ns" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster basic-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns, error: failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-vcq5ns\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0016c03a0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns",
            },
            <*fmt.wrapErrors | 0xc0016e5170>{
                msg: "failed waiting for cluster=\"basic-cluster\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-vcq5ns\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0016c03a0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-vcq5ns",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Cluster should become ready within 45 minutes
17:04:22 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:100]: Unexpected error:
    <*fmt.wrapErrors | 0xc00110c270>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-4mjqcf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4mjqcf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e7da60>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf",
            },
            <*fmt.wrapErrors | 0xc00110c240>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4mjqcf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e7da60>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:100]: Unexpected error:
    <*fmt.wrapErrors | 0xc00110c270>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-4mjqcf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4mjqcf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e7da60>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf",
            },
            <*fmt.wrapErrors | 0xc00110c240>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-4mjqcf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e7da60>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-4mjqcf",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
    <*fmt.wrapErrors | 0xc00119f3b0>: 
    failed to create HCP cluster list-test-cluster-6m2xvx, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x, error: failed waiting for cluster="list-test-cluster-6m2xvx" in resourcegroup="cluster-listing-27cs4x" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster list-test-cluster-6m2xvx, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x, error: failed waiting for cluster=\"list-test-cluster-6m2xvx\" in resourcegroup=\"cluster-listing-27cs4x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000d7b490>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x",
            },
            <*fmt.wrapErrors | 0xc00119f380>{
                msg: "failed waiting for cluster=\"list-test-cluster-6m2xvx\" in resourcegroup=\"cluster-listing-27cs4x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000d7b490>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:86]: Unexpected error:
    <*fmt.wrapErrors | 0xc00119f3b0>: 
    failed to create HCP cluster list-test-cluster-6m2xvx, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x, error: failed waiting for cluster="list-test-cluster-6m2xvx" in resourcegroup="cluster-listing-27cs4x" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster list-test-cluster-6m2xvx, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x, error: failed waiting for cluster=\"list-test-cluster-6m2xvx\" in resourcegroup=\"cluster-listing-27cs4x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000d7b490>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x",
            },
            <*fmt.wrapErrors | 0xc00119f380>{
                msg: "failed waiting for cluster=\"list-test-cluster-6m2xvx\" in resourcegroup=\"cluster-listing-27cs4x\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000d7b490>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster list-test-cluster-6m2xvx in resource group cluster-listing-27cs4x",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc00120b2c0>: 
    failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-mn7k92" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92, error: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mn7k92\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0013e3bc0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92",
            },
            <*fmt.wrapErrors | 0xc00120b290>{
                msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mn7k92\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0013e3bc0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc00120b2c0>: 
    failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-mn7k92" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92, error: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mn7k92\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0013e3bc0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92",
            },
            <*fmt.wrapErrors | 0xc00120b290>{
                msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mn7k92\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0013e3bc0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-mn7k92",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e88de0>: 
    failed to create HCP cluster ea-listcng8jl, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d, error: failed waiting for cluster="ea-listcng8jl" in resourcegroup="ea-list-f4k46d" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-listcng8jl, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d, error: failed waiting for cluster=\"ea-listcng8jl\" in resourcegroup=\"ea-list-f4k46d\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009b7af0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d",
            },
            <*fmt.wrapErrors | 0xc000e88db0>{
                msg: "failed waiting for cluster=\"ea-listcng8jl\" in resourcegroup=\"ea-list-f4k46d\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0009b7af0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e88de0>: 
    failed to create HCP cluster ea-listcng8jl, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d, error: failed waiting for cluster="ea-listcng8jl" in resourcegroup="ea-list-f4k46d" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-listcng8jl, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d, error: failed waiting for cluster=\"ea-listcng8jl\" in resourcegroup=\"ea-list-f4k46d\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009b7af0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d",
            },
            <*fmt.wrapErrors | 0xc000e88db0>{
                msg: "failed waiting for cluster=\"ea-listcng8jl\" in resourcegroup=\"ea-list-f4k46d\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0009b7af0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listcng8jl in resource group ea-list-f4k46d",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc00158db00>: 
    failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-v98q2r" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-v98q2r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011878f0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r",
            },
            <*fmt.wrapErrors | 0xc00158dad0>{
                msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-v98q2r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011878f0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:91]: Unexpected error:
    <*fmt.wrapErrors | 0xc00158db00>: 
    failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r, error: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-v98q2r" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster sre-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r, error: failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-v98q2r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011878f0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r",
            },
            <*fmt.wrapErrors | 0xc00158dad0>{
                msg: "failed waiting for cluster=\"sre-hcp-cluster\" in resourcegroup=\"admin-api-breakglass-v98q2r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011878f0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster sre-hcp-cluster in resource group admin-api-breakglass-v98q2r",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Interrupted by User
17:04:22 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc001053530>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-f5ctsq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-f5ctsq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fbd7d0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq",
            },
            <*fmt.wrapErrors | 0xc001053500>{
                msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-f5ctsq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fbd7d0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapErrors | 0xc001053530>: 
    failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq, error: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-f5ctsq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq, error: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-f5ctsq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fbd7d0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq",
            },
            <*fmt.wrapErrors | 0xc001053500>{
                msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-f5ctsq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fbd7d0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-cluster in resource group external-auth-cluster-f5ctsq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d8ccf0>: 
    failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-ct6m25" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-ct6m25\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010460b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25",
            },
            <*fmt.wrapErrors | 0xc000d8ccc0>{
                msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-ct6m25\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010460b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:87]: Unexpected error:
    <*fmt.wrapErrors | 0xc000d8ccf0>: 
    failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25, error: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-ct6m25" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster engineering-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25, error: failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-ct6m25\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010460b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25",
            },
            <*fmt.wrapErrors | 0xc000d8ccc0>{
                msg: "failed waiting for cluster=\"engineering-hcp-cluster\" in resourcegroup=\"kusto-logs-ct6m25\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0010460b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster engineering-hcp-cluster in resource group kusto-logs-ct6m25",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Deserializaion Error: no output from command
17:04:22 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
progress report:
      Customer should respect cluster-wide node limits with nodepool autoscaling (Spec Runtime: 1h29m59.829s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122
        In [It] (Node Runtime: 1h29m59.829s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122
          At [By Step] creating cluster with MaxNodesTotal limit (Step Runtime: 45m53.738s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:143

          Spec Goroutine
          goroutine 100 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x820d118, 0xc000ab84b0}, 0x77f2f7b?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x823ff60, {0x820d118, 0xc000ab84b0}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x820d188?, 0xc0001ee2a0?}, {{0x82228a8?, 0xc00088d280?}, 0xc001450970?}, 0xc000ff5810, {0xc00023fa70, 0x16}, {0x780eccd, 0x12}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:641
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77d3694, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func23.3({0x7fcf776a4250, 0xc0009062a0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:160
                | Expect(err).NotTo(HaveOccurred())
                | 
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8222980?, 0xc0009062a0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 65
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166 @ 03/05/26 19:51:01.88
  "ts"="2026-03-05 19:51:01.880734" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-05 19:51:01.880823" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-node-limit-cluster-fphxbn" "resourceGroup"="node-limit-test-plc8qm"
  "ts"="2026-03-05 19:51:02.813337" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-node-limit-cluster-fphxbn" "resourceGroup"=""
  "ts"="2026-03-05 19:51:03.675003" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz"
  "ts"="2026-03-05 19:51:03.675102" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz"
  "ts"="2026-03-05 19:51:03.675170" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-05 19:51:03.676249" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-204"
  "ts"="2026-03-05 19:51:03.676262" "level"=0 "msg"="collecting deployments" "resourceGroup"="node-limit-test-plc8qm"
  "ts"="2026-03-05 19:51:04.225946" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-204" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-05 19:51:04.623087" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-plc8qm" "deployment"="pooledNonMsiScopedAssignments-j6nw3ohx7bzo4"
  "ts"="2026-03-05 19:51:04.982469" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-204" "deployment"="msi-deployment-204"
  "ts"="2026-03-05 19:51:05.382145" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-plc8qm" "deployment"="customer-infra-node-limit-cluster-fphxbn"
  "ts"="2026-03-05 19:51:06.229898" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-05 19:51:06.230046" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-05 19:51:06.230202" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="node-limit-test-plc8qm"
Command Error: signal: killed
Deserializaion Error: no output from command
progress report:
      Customer should respect cluster-wide node limits with nodepool autoscaling (Spec Runtime: 1h29m59.829s)
        /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122
        In [It] (Node Runtime: 1h29m59.829s)
          /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:122
          At [By Step] creating cluster with MaxNodesTotal limit (Step Runtime: 45m53.738s)
            /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:143

          Spec Goroutine
          goroutine 100 [select]
            github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x820d118, 0xc000ab84b0}, 0x77f2f7b?)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36
            github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x823ff60, {0x820d118, 0xc000ab84b0}, 0x0)
              /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284
            github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x820d188?, 0xc0001ee2a0?}, {{0x82228a8?, 0xc00088d280?}, 0xc001450970?}, 0xc000ff5810, {0xc00023fa70, 0x16}, {0x780eccd, 0x12}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:641
            github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x77d3694, 0x4}, ...}, ...)
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:344
          > github.com/Azure/ARO-HCP/test/e2e.init.func23.3({0x7fcf776a4250, 0xc0009062a0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:160
                | Expect(err).NotTo(HaveOccurred())
                | 
                > err = tc.CreateHCPClusterFromParam(ctx,
                | 	GinkgoLogr,
                | 	*resourceGroup.Name,
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8222980?, 0xc0009062a0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 65
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166 @ 03/05/26 19:51:01.88
  "ts"="2026-03-05 19:51:01.880734" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-05 19:51:01.880823" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-node-limit-cluster-fphxbn" "resourceGroup"="node-limit-test-plc8qm"
  "ts"="2026-03-05 19:51:02.813337" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-node-limit-cluster-fphxbn" "resourceGroup"=""
  "ts"="2026-03-05 19:51:03.675003" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz"
  "ts"="2026-03-05 19:51:03.675102" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-133f4b3b85cd881bd2d7025c4df814bf698b577d4462803137df4f5ac0cf68e9.yaml.gz"
  "ts"="2026-03-05 19:51:03.675170" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-05 19:51:03.676249" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-204"
  "ts"="2026-03-05 19:51:03.676262" "level"=0 "msg"="collecting deployments" "resourceGroup"="node-limit-test-plc8qm"
  "ts"="2026-03-05 19:51:04.225946" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-204" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-05 19:51:04.623087" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-plc8qm" "deployment"="pooledNonMsiScopedAssignments-j6nw3ohx7bzo4"
  "ts"="2026-03-05 19:51:04.982469" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-204" "deployment"="msi-deployment-204"
  "ts"="2026-03-05 19:51:05.382145" "level"=0 "msg"="collecting operations" "resourceGroup"="node-limit-test-plc8qm" "deployment"="customer-infra-node-limit-cluster-fphxbn"
  "ts"="2026-03-05 19:51:06.229898" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-05 19:51:06.230046" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-05 19:51:06.230202" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="node-limit-test-plc8qm"
Command Error: signal: killed
Deserializaion Error: no output from command
Deserializaion Error: no output from command
17:04:22 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel
Show raw failure
:284
          > github.com/Azure/ARO-HCP/test/e2e.init.func16.2({0x7f9950e14260, 0xc00071d920})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
                | 
                | By("waiting for first cluster to complete creation")
                > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
                | 	Frequency: framework.StandardPollInterval,
                | })
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8222980?, 0xc00071d920?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 91
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/05/26 19:51:01.879
  "ts"="2026-03-05 19:51:01.879639" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-05 19:51:01.879740" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-wdnsms" "resourceGroup"="customer-rg-z2vc29"
  "ts"="2026-03-05 19:51:02.856126" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-wdnsms" "resourceGroup"=""
  "ts"="2026-03-05 19:51:03.876876" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-lfq9wm" "resourceGroup"="customer-rg-z2vc29"
  "ts"="2026-03-05 19:51:04.021245" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-lfq9wm" "resourceGroup"=""
  "ts"="2026-03-05 19:51:04.831197" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-05 19:51:04.831315" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-05 19:51:04.831390" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-05 19:51:04.833064" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-160"
  "ts"="2026-03-05 19:51:04.833097" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-91"
  "ts"="2026-03-05 19:51:04.833131" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-z2vc29"
  "ts"="2026-03-05 19:51:04.833088" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-297"
  "ts"="2026-03-05 19:51:05.359625" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-91" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-05 19:51:05.745001" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-160" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-05 19:51:06.201250" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-z2vc29" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
  "ts"="2026-03-05 19:51:06.623151" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-297" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-05 19:51:06.990038" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-91" "deployment"="msi-deployment-91"
  "ts"="2026-03-05 19:51:07.388213" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-160" "deployment"="msi-deployment-160"
  "ts"="2026-03-05 19:51:07.781171" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-z2vc29" "deployment"="customer-infra-basic-hcp-cluster2-lfq9wm"
  "ts"="2026-03-05 19:51:08.151736" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-297" "deployment"="msi-deployment-297"
  "ts"="2026-03-05 19:51:09.473452" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-z2vc29" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
  "ts"="2026-03-05 19:51:10.117976" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-z2vc29" "deployment"="customer-infra-basic-hcp-cluster-wdnsms"
  "ts"="2026-03-05 19:51:10.638792" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-05 19:51:10.638932" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-05 19:51:10.639040" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-z2vc29"
Command Error: signal: killed
Deserializaion Error: no output from command
:284
          > github.com/Azure/ARO-HCP/test/e2e.init.func16.2({0x7f9950e14260, 0xc00071d920})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
                | 
                | By("waiting for first cluster to complete creation")
                > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
                | 	Frequency: framework.StandardPollInterval,
                | })
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x8222980?, 0xc00071d920?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 91
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/05/26 19:51:01.879
  "ts"="2026-03-05 19:51:01.879639" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-05 19:51:01.879740" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-wdnsms" "resourceGroup"="customer-rg-z2vc29"
  "ts"="2026-03-05 19:51:02.856126" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-wdnsms" "resourceGroup"=""
  "ts"="2026-03-05 19:51:03.876876" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-lfq9wm" "resourceGroup"="customer-rg-z2vc29"
  "ts"="2026-03-05 19:51:04.021245" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-lfq9wm" "resourceGroup"=""
  "ts"="2026-03-05 19:51:04.831197" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-05 19:51:04.831315" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-05 19:51:04.831390" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-05 19:51:04.833064" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-160"
  "ts"="2026-03-05 19:51:04.833097" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-91"
  "ts"="2026-03-05 19:51:04.833131" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-z2vc29"
  "ts"="2026-03-05 19:51:04.833088" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-dev-297"
  "ts"="2026-03-05 19:51:05.359625" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-91" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-05 19:51:05.745001" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-160" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-05 19:51:06.201250" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-z2vc29" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
  "ts"="2026-03-05 19:51:06.623151" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-297" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-05 19:51:06.990038" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-91" "deployment"="msi-deployment-91"
  "ts"="2026-03-05 19:51:07.388213" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-160" "deployment"="msi-deployment-160"
  "ts"="2026-03-05 19:51:07.781171" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-z2vc29" "deployment"="customer-infra-basic-hcp-cluster2-lfq9wm"
  "ts"="2026-03-05 19:51:08.151736" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-dev-297" "deployment"="msi-deployment-297"
  "ts"="2026-03-05 19:51:09.473452" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-z2vc29" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
  "ts"="2026-03-05 19:51:10.117976" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-z2vc29" "deployment"="customer-infra-basic-hcp-cluster-wdnsms"
  "ts"="2026-03-05 19:51:10.638792" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-05 19:51:10.638932" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-05 19:51:10.639040" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-z2vc29"
Command Error: signal: killed
Deserializaion Error: no output from command
timeout during CreateHCPClusterAndWait; context deadline exceeded
17:04:22 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc000163740>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-lk2bm6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-lk2bm6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001098820>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6",
            },
            <*fmt.wrapErrors | 0xc000163710>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-lk2bm6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001098820>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc000163740>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-lk2bm6" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-lk2bm6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001098820>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6",
            },
            <*fmt.wrapErrors | 0xc000163710>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-lk2bm6\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001098820>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lk2bm6",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7cbe20c · merge 7cbe20c
 n/a n/a passed #4218 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4c15703 · merge 4c15703
 n/a n/a passed #4075 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9658e7f
 n/a n/a passed #4330 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 26b2f21 · merge 26b2f21
 n/a provision failed #4062 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
15:19:20 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T15:56:11.735Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T15:56:13.671Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T15:56:14.913Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=ed84426758cbbb353371a9235a496758b8f3e98aa625c265db09a4e3e29cda2c
time=2026-03-05T15:58:15.933Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7455488/providers/Microsoft.Resources/deployments/ed84426758cbbb353371a9235a496758b8f3e98aa625c265db09a4e3e29cda2c/operationStatuses/08584288807112291690\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7455488/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7455488/providers/Microsoft.Monitor/accounts/hcps-j7455488/providers/Microsoft.Authorization/roleAssignments/caa896f1-1d74-5c65-824e-d46f9b44719f\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T155635Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '6e38dbe3-68e4-4adb-b4dc-72161e062672', request correlation id '1056ad0c-dd70-4639-a07e-4effd484f439'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7455488/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7455488/providers/Microsoft.Monitor/accounts/services-j7455488/providers/Microsoft.Authorization/roleAssignments/fb67040b-742c-5416-bfc1-eb6a3205318d\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T155628Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '83a7fc85-9a1f-4ce7-a764-3f5342de9960', request correlation id '1056ad0c-dd70-4639-a07e-4effd484f439'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5501d1c
 n/a provision failed #4333 (closed) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
15:16:58 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T15:54:30.586Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T15:54:32.976Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T15:54:34.248Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=a89c2c923545ab2245aae3e95f8c1e0e15fd41c82d7aa26d45863bc410cd51f2
time=2026-03-05T15:56:35.445Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j4296704/providers/Microsoft.Resources/deployments/a89c2c923545ab2245aae3e95f8c1e0e15fd41c82d7aa26d45863bc410cd51f2/operationStatuses/08584288808119232283\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4296704/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j4296704/providers/Microsoft.Monitor/accounts/services-j4296704/providers/Microsoft.Authorization/roleAssignments/f804a3aa-59ee-5d3f-8e21-9cd520cda0fb\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T155449Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'a38e6fcf-aa35-42d4-ba05-70159e9be81c', request correlation id 'a8ffd35f-99a4-4c58-952a-d6ac85d98793'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 26b2f21 · merge 26b2f21
 n/a n/a passed #4062 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 3f1edac · merge 77248e3
 n/a n/a passed #4334 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0ac5e19 · merge 0ac5e19
 n/a n/a passed #4181 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 87ae44a · merge 87ae44a
 n/a n/a passed #4154 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 24e90da
 n/a provision failed #4117 (closed) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
13:09:31 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T13:52:21.471Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T13:52:23.729Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T13:52:24.922Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=968184862e8d8e1e198740f9cb9a18dbb9f3ffea5f2abd8e139d50c47a886982
time=2026-03-05T13:54:26.034Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j3809152/providers/Microsoft.Resources/deployments/968184862e8d8e1e198740f9cb9a18dbb9f3ffea5f2abd8e139d50c47a886982/operationStatuses/08584288881413094625\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3809152/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j3809152/providers/Microsoft.Monitor/accounts/services-j3809152/providers/Microsoft.Authorization/roleAssignments/2b74f53a-392c-5475-8b8f-233c5419fcce\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T135240Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '2019e909-eeef-4b4d-8f2c-c845885ac574', request correlation id '274aca8f-9e3b-4890-b7a6-a767e9f62905'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9658e7f
 n/a n/a passed #4330 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head b20d783
 n/a provision failed #4332 (closed) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
12:37:25 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T13:19:53.786Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T13:19:55.939Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T13:19:57.296Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=885a04aacb6c94df7f66dc150664bc8a0d8e5af210e4f66e64290e9845f84213
time=2026-03-05T13:21:58.139Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2267264/providers/Microsoft.Resources/deployments/885a04aacb6c94df7f66dc150664bc8a0d8e5af210e4f66e64290e9845f84213/operationStatuses/08584288900888641821\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2267264/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2267264/providers/Microsoft.Monitor/accounts/services-j2267264/providers/Microsoft.Authorization/roleAssignments/57c394cb-bdfc-5798-a547-862f585456b9\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T132028Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '34a0ff1d-d958-4980-bd2b-0c329b20f2f3', request correlation id 'fd649a6d-c292-475c-ae6b-80e1d0ea8641'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 28818a1 · merge 28818a1
 n/a n/a passed #4331 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4b5ae23 · merge 4b5ae23
 n/a n/a passed #4310 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 4b5ae23 · merge 4b5ae23
 n/a provision failed #4310 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
12:08:28 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T12:38:31.853Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T12:38:34.079Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T12:38:36.335Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=732ba973089226b8b663a9106b6bf45de4247584339dbd77fed8a16eeefb62ad
time=2026-03-05T12:40:37.249Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j7740672/providers/Microsoft.Resources/deployments/732ba973089226b8b663a9106b6bf45de4247584339dbd77fed8a16eeefb62ad/operationStatuses/08584288925702979837\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7740672/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j7740672/providers/Microsoft.Monitor/accounts/hcps-j7740672/providers/Microsoft.Authorization/roleAssignments/17ef7e10-833d-58a0-a0e2-baa16cfead1c\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T123858Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b9755a63-bf9f-45fa-8f71-f6f909b7ec9d', request correlation id 'a972a4cc-ad02-4d23-bfb7-980913db976a'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head fc6aeff · merge 78f5869
 n/a provision failed #3838 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
12:03:51 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T12:38:31.400Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T12:38:33.901Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T12:38:35.030Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=7deaf6dd6bfd1ba1e9a85282e492585ecece9886555a54060f37a3386193663e
time=2026-03-05T12:40:35.984Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5361280/providers/Microsoft.Resources/deployments/7deaf6dd6bfd1ba1e9a85282e492585ecece9886555a54060f37a3386193663e/operationStatuses/08584288925711000171\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5361280/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5361280/providers/Microsoft.Monitor/accounts/hcps-j5361280/providers/Microsoft.Authorization/roleAssignments/34f079ee-4416-5c6d-9cc3-7033731276bc\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T123847Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '159f92a4-2a32-4434-8a80-3a4229adc247', request correlation id 'dce5004b-b69c-44fa-8a34-0a52c348965a'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
11:39:39 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T11:42:36.822Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T11:42:39.265Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T11:42:40.557Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=4b844f63aabbc275696849fa33c8691b6a9b9b2f2da80229ad2e6ece8380cd88
time=2026-03-05T11:46:42.111Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j2734208/providers/Microsoft.Resources/deployments/4b844f63aabbc275696849fa33c8691b6a9b9b2f2da80229ad2e6ece8380cd88/operationStatuses/08584288959256726696\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2734208/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2734208/providers/Microsoft.Monitor/accounts/hcps-j2734208/providers/Microsoft.Authorization/roleAssignments/22b2f941-137e-5fbb-899f-102955913afc\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T114255Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '6fc9826d-0897-483e-b974-17a8606ac75c', request correlation id '441331fa-02ce-4484-a005-f0a42898d604'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2734208/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j2734208/providers/Microsoft.Monitor/accounts/services-j2734208/providers/Microsoft.Authorization/roleAssignments/72b9a98c-bbb2-585a-a5bd-43bc46b2e197\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T114311Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '7c09b509-8240-4f47-94bd-c4d8bfca9e79', request correlation id '441331fa-02ce-4484-a005-f0a42898d604'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 33bde46 · merge 33bde46
 n/a provision failed #4243 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
10:47:55 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T11:17:36.021Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T11:17:38.376Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T11:17:39.730Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=dcdf53ab6a33f4648a45c10412b552aac5dc39f0a0b46cfd24d0996447c79fd8
time=2026-03-05T11:19:40.745Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j5892224/providers/Microsoft.Resources/deployments/dcdf53ab6a33f4648a45c10412b552aac5dc39f0a0b46cfd24d0996447c79fd8/operationStatuses/08584288974266074851\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5892224/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j5892224/providers/Microsoft.Monitor/accounts/hcps-j5892224/providers/Microsoft.Authorization/roleAssignments/3822e227-7632-564b-a3b3-d728937b0fe9\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T111801Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'b398975b-1bbb-4c1c-9400-f87cf78f2c8f', request correlation id '7a55ace7-095f-4e18-91c6-86114a7a425e'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head a88a31f · merge a88a31f
 n/a n/a passed #4316 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9658e7f
 n/a n/a passed #4330 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head dfe5ac7 · merge 0289e5c
 n/a n/a passed #4329 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0bc9efd · merge 0bc9efd
 n/a n/a passed #4320 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4b5ae23 · merge 4b5ae23
 n/a n/a passed #4310 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 33bde46 · merge 33bde46
 n/a e2e failed #4243 (merged) 1
e2e (1)
failed to create route: the server is currently unable to handle the request
08:41:19 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:131]: Unexpected error:
    <*fmt.wrapError | 0xc00041a960>: 
    failed to create route: the server is currently unable to handle the request
    {
        msg: "failed to create route: the server is currently unable to handle the request",
        err: <*errors.StatusError | 0xc000848000>{
            ErrStatus: {
                TypeMeta: {Kind: "", APIVersion: ""},
                ListMeta: {
                    SelfLink: "",
                    ResourceVersion: "",
                    Continue: "",
                    RemainingItemCount: nil,
                },
                Status: "Failure",
                Message: "the server is currently unable to handle the request",
                Reason: "ServiceUnavailable",
                Details: {
                    Name: "",
                    Group: "",
                    Kind: "",
                    UID: "",
                    Causes: [
                        {
                            Type: "UnexpectedServerResponse",
                            Message: "error trying to reach service: EOF",
                            Field: "",
                        },
                    ],
                    RetryAfterSeconds: 0,
                },
                Code: 503,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:131]: Unexpected error:
    <*fmt.wrapError | 0xc00041a960>: 
    failed to create route: the server is currently unable to handle the request
    {
        msg: "failed to create route: the server is currently unable to handle the request",
        err: <*errors.StatusError | 0xc000848000>{
            ErrStatus: {
                TypeMeta: {Kind: "", APIVersion: ""},
                ListMeta: {
                    SelfLink: "",
                    ResourceVersion: "",
                    Continue: "",
                    RemainingItemCount: nil,
                },
                Status: "Failure",
                Message: "the server is currently unable to handle the request",
                Reason: "ServiceUnavailable",
                Details: {
                    Name: "",
                    Group: "",
                    Kind: "",
                    UID: "",
                    Causes: [
                        {
                            Type: "UnexpectedServerResponse",
                            Message: "error trying to reach service: EOF",
                            Field: "",
                        },
                    ],
                    RetryAfterSeconds: 0,
                },
                Code: 503,
            },
        },
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 3832c3f · merge 3832c3f
 n/a n/a passed #4317 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a88a31f · merge a88a31f
 n/a n/a passed #4316 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 79cc087 · merge 79cc087
 n/a n/a passed #4323 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 55a4b56 · merge 55a4b56
 n/a provision failed #4328 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
07:37:07 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T08:08:56.518Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T08:08:58.744Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T08:08:59.930Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=f0307857dfa4bfd4a491c13874b3f35a2260078b15aa1ecf510bb1d6073c3c09
time=2026-03-05T08:11:01.056Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j6759040/providers/Microsoft.Resources/deployments/f0307857dfa4bfd4a491c13874b3f35a2260078b15aa1ecf510bb1d6073c3c09/operationStatuses/08584289087462844066\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6759040/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6759040/providers/Microsoft.Monitor/accounts/services-j6759040/providers/Microsoft.Authorization/roleAssignments/93fe7d5e-4700-5dbe-8042-339e6c44c3d2\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T080915Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '8d1df159-6531-446c-92df-b32aa73dbf86', request correlation id 'f4290445-2b26-4174-b5ae-b105dcb1b60e'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6759040/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j6759040/providers/Microsoft.Monitor/accounts/hcps-j6759040/providers/Microsoft.Authorization/roleAssignments/c38e841b-728d-55f7-9bac-de32bb8c12bd\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T080915Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'da9e525b-7158-4fe6-981e-d2f3887819b0', request correlation id 'f4290445-2b26-4174-b5ae-b105dcb1b60e'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 99ea3cd · merge efa8e74
 n/a n/a passed #4279 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head b61beb9 · merge b61beb9
 n/a provision failed #4285 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
06:40:21 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T07:10:22.878Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T07:10:24.806Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T07:10:26.338Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=08417ab41b55b6beab458d8c23ff58bc494581a90ac44125e29dc35ecb4d5e8a
time=2026-03-05T07:11:57.185Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8614784/providers/Microsoft.Resources/deployments/08417ab41b55b6beab458d8c23ff58bc494581a90ac44125e29dc35ecb4d5e8a/operationStatuses/08584289122600834882\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8614784/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8614784/providers/Microsoft.Monitor/accounts/services-j8614784/providers/Microsoft.Authorization/roleAssignments/3698ffb6-8a5b-59a1-8ae2-7381805860f7\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T071041Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '36ff17e3-ab53-4bec-8d0d-a4320a4e9aa4', request correlation id 'e6106325-27db-4264-8ec8-a9bd301d3453'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7148a34 · merge 7148a34
 n/a n/a passed #4303 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 7148a34 · merge 7148a34
 n/a n/a passed #4303 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4b5ae23 · merge 4b5ae23
 n/a n/a passed #4310 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 79cc087 · merge 79cc087
 n/a n/a passed #4323 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 6e48d06 · merge 6e48d06
 n/a n/a passed #4318 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 507eb19 · merge 507eb19
 n/a provision failed #4290 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
02:08:36 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T02:44:42.127Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T02:44:44.225Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T02:44:45.388Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=c33883562cbb0565bbc4b8b1461471b8d2373275b5386b7f0dbfdd020fee5a8c
time=2026-03-05T02:46:46.414Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8410624/providers/Microsoft.Resources/deployments/c33883562cbb0565bbc4b8b1461471b8d2373275b5386b7f0dbfdd020fee5a8c/operationStatuses/08584289282008278246\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8410624/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8410624/providers/Microsoft.Monitor/accounts/services-j8410624/providers/Microsoft.Authorization/roleAssignments/1fea680a-204e-57e9-abe6-5bbd387eb69b\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T024511Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id 'ef02a3fa-f694-498a-8dd4-d48df935563a', request correlation id 'f9ce2c32-e549-4f79-b85b-46cd377186a5'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7148a34 · merge 7148a34
 n/a n/a passed #4303 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 7148a34 · merge 7148a34
 n/a provision failed #4303 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
00:54:11 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T01:23:56.332Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T01:23:58.035Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T01:23:59.120Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=a01796ac4baa46ceaa0df83ddcb8473965a70bce3dce9cfa70dc64582e1b4346
time=2026-03-05T01:25:30.099Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8561408/providers/Microsoft.Resources/deployments/a01796ac4baa46ceaa0df83ddcb8473965a70bce3dce9cfa70dc64582e1b4346/operationStatuses/08584289330470171432\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8561408/providers/Microsoft.Resources/deployments/svc-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8561408/providers/Microsoft.Monitor/accounts/services-j8561408/providers/Microsoft.Authorization/roleAssignments/ea66a137-5ba7-5402-bca5-53fef663ece7\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T012421Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '4bd23607-a440-4e74-8dc6-f4c1beca3ab3', request correlation id 'af7d6d88-23e0-4957-a2fc-5705c4fc29df'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8561408/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8561408/providers/Microsoft.Monitor/accounts/hcps-j8561408/providers/Microsoft.Authorization/roleAssignments/c2a8e746-a51c-5395-a742-9764820801d4\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T012413Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '6d2109f9-3697-4256-9aac-f890d6a4f099', request correlation id 'af7d6d88-23e0-4957-a2fc-5705c4fc29df'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 9bce19c · merge efa8e74
 n/a provision failed #4279 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.EventGrid
00:13:42 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T00:51:55.489Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T00:51:57.551Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T00:51:58.689Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=e289e452ce30dca8b9b33fbec9a710150ba61ec8edecfa9d7f9b5a5bb2ee010c
time=2026-03-05T00:53:29.477Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j1043456/providers/Microsoft.Resources/deployments/e289e452ce30dca8b9b33fbec9a710150ba61ec8edecfa9d7f9b5a5bb2ee010c/operationStatuses/08584289349674868518\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j1043456/providers/Microsoft.Resources/deployments/maestro-infra-deployment\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j1043456/providers/Microsoft.EventGrid/namespaces/arohcp-prow-maestro-j1043456\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"Internal error\\\",\\r\\n                \\\"message\\\": \\\"The operation failed due to an internal server error. The initial state of the impacted resources (if any) are restored. Please try again in few minutes. If error still persists, report 87f09325-e80e-49f7-81d9-4e95395939c8:3/5/2026 12:53:06 AM (UTC) to our forums for assistance or raise a support ticket .\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 9bce19c · merge efa8e74
 n/a provision failed #4279 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code Conflict; provider Microsoft.Authorization
00:10:52 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Region/regional/infra · suite step graph
Show raw failure
time=2026-03-05T00:41:38.724Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-03-05T00:41:41.203Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-03-05T00:41:42.300Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=d2d5197b55c6d9d0e84f8794e76ca7d3393df4f5fca5fc6f8f13198c5f7020fc
time=2026-03-05T00:43:13.044Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-prow-j8957824/providers/Microsoft.Resources/deployments/d2d5197b55c6d9d0e84f8794e76ca7d3393df4f5fca5fc6f8f13198c5f7020fc/operationStatuses/08584289355838782821\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8957824/providers/Microsoft.Resources/deployments/hcp-monitor\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-prow-j8957824/providers/Microsoft.Monitor/accounts/hcps-j8957824/providers/Microsoft.Authorization/roleAssignments/b78ed5ca-6bf2-5a8b-afef-8067513778d3\\\",\\r\\n            \\\"message\\\": \\\"Encountered internal server error. Diagnostic information: timestamp '20260305T004157Z', subscription id 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX', tracking id '048ab58c-9aac-4ae5-ac20-57059a42ce03', request correlation id 'c8b1f931-df64-4ba1-bd53-847f7f697a0b'.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"