<
Single Day: Mar 12
> Reset
View JSON API

Environment: INT

Failure patterns: 8 · Failures matched: 21 · Total runs: 9 · Runs affected: 6

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: LockContention; detail message Failed to acquire lock: timed out; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e222.22%2 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsMar 6: 0 · Mar 7: 0 · Mar 8: 0 · Mar 9: 0 · Mar 10: 1 · Mar 11: 0 · Mar 12: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error:
    <*fmt.wrapError | 0xc00038c420>: 
    failed creating external auth "ea-list" in resourcegroup="ea-list-9lhb8x" for cluster="ea-listgv59sm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9lhb8x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listgv59sm/externalAuths/ea-list
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-9lhb8x\" for cluster=\"ea-listgv59sm\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9lhb8x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listgv59sm/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00076e300>{
            ErrorCode: "LockContention",
            StatusCode: 503,
            RawResponse: {
                Status: "503 Service Unavailable",
                StatusCode: 503,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Retry-After": ["10"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/4616b9c4-eae5-42c0-b42e-e60136e4b27a",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Thu, 12 Mar 2026 22:53:06 GMT",
                    ],
                    "Content-Length": ["186"],
                    "X-Ms-Request-Id": [
                        "0daaf6b9-dea5-419a-a781-f5d3f0c88c48",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": ["LockContention"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: DD91E82BD13A4C8F974F7F6335BA4D4F Ref B: CHI261140506025 Ref C: 2026-03-12T22:52:57Z",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "X-Envoy-Upstream-Service-Time": ["10007"],
                    "X-Ms-Correlation-Request-Id": [
                        "132f0482-3ab7-4255-98f8-45a73cf0c24c",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260312T225307Z:132f0482-3ab7-4255-98f8-45a73cf0c24c",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00038c200>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 101, 98, 99, 50, 45, 52, 49, 53, 54, 45, 57, 100, 57, 49, 45, 99, 52, 99, 55, 56, 49, 100, 101, 55, 101, 53, 52, 34, 10, 32, 32, 32, 32, 125, 10, 125]...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:118]: Unexpected error:
    <*fmt.wrapError | 0xc00038c420>: 
    failed creating external auth "ea-list" in resourcegroup="ea-list-9lhb8x" for cluster="ea-listgv59sm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9lhb8x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listgv59sm/externalAuths/ea-list
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed creating external auth \"ea-list\" in resourcegroup=\"ea-list-9lhb8x\" for cluster=\"ea-listgv59sm\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-9lhb8x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listgv59sm/externalAuths/ea-list\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc00076e300>{
            ErrorCode: "LockContention",
            StatusCode: 503,
            RawResponse: {
                Status: "503 Service Unavailable",
                StatusCode: 503,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Retry-After": ["10"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/4616b9c4-eae5-42c0-b42e-e60136e4b27a",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Thu, 12 Mar 2026 22:53:06 GMT",
                    ],
                    "Content-Length": ["186"],
                    "X-Ms-Request-Id": [
                        "0daaf6b9-dea5-419a-a781-f5d3f0c88c48",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": ["LockContention"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: DD91E82BD13A4C8F974F7F6335BA4D4F Ref B: CHI261140506025 Ref C: 2026-03-12T22:52:57Z",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Content-Type": ["application/json"],
                    "X-Envoy-Upstream-Service-Time": ["10007"],
                    "X-Ms-Correlation-Request-Id": [
                        "132f0482-3ab7-4255-98f8-45a73cf0c24c",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260312T225307Z:132f0482-3ab7-4255-98f8-45a73cf0c24c",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc00038c200>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 76, 111, 99, 107, 67, 111, 110, 116, 101, 110, 116, 105, 111, 110, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 70, 97, 105, 108, 101, 100, 32, 116, 111, 32, 97, 99, 113, 117, 105, 114, 101, 32, 108, 111, 99, 107, 58, 32, 116, 105, 109, 101, 100, 32, 111, 117, 116, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 116, 97, 114, 103, 101, 116, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 101, 98, 99, 50, 45, 52, 49, 53, 54, 45, 57, 100, 57, 49, 45, 99, 52, 99, 55, 56, 49, 100, 101, 55, 101, 53, 52, 34, 10, 32, 32, 32, 32, 125, 10, 125]...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
    <*fmt.wrapError | 0xc00148e400>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-h2fml7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-h2fml7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00148e3e0>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-h2fml7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000b78690>{
                ErrorCode: "LockContention",
                StatusCode: 503,
                RawResponse: {
                    Status: "503 Service Unavailable",
                    StatusCode: 503,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Content-Length": ["186"],
                        "X-Ms-Error-Code": ["LockContention"],
                        "X-Envoy-Upstream-Service-Time": ["10025"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/ab289601-a1be-4eea-baec-7844c0a021f7",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Thu, 12 Mar 2026 22:53:06 GMT",
                        ],
                        "Content-Type": ["application/json"],
                        "Retry-After": ["10"],
                        "X-Ms-Failure-Cause": ["service"],
                        "X-Msedge-Ref": [
                            "Ref A: 6E495E82722B47FAB217CA309B5EFC10 Ref B: CHI261140501060 Ref C: 2026-03-12T22:52:57Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "c86fd44b-9640-4399-b837-438b4fc03302",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "8be5b8ee-e739-4b49-a85a-b05296aaa904",
                        ],
               ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:145]: Unexpected error:
    <*fmt.wrapError | 0xc00148e400>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-h2fml7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-h2fml7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00148e3e0>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-h2fml7\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-h2fml7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000b78690>{
                ErrorCode: "LockContention",
                StatusCode: 503,
                RawResponse: {
                    Status: "503 Service Unavailable",
                    StatusCode: 503,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Content-Length": ["186"],
                        "X-Ms-Error-Code": ["LockContention"],
                        "X-Envoy-Upstream-Service-Time": ["10025"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/ab289601-a1be-4eea-baec-7844c0a021f7",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Thu, 12 Mar 2026 22:53:06 GMT",
                        ],
                        "Content-Type": ["application/json"],
                        "Retry-After": ["10"],
                        "X-Ms-Failure-Cause": ["service"],
                        "X-Msedge-Ref": [
                            "Ref A: 6E495E82722B47FAB217CA309B5EFC10 Ref B: CHI261140501060 Ref C: 2026-03-12T22:52:57Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "c86fd44b-9640-4399-b837-438b4fc03302",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "8be5b8ee-e739-4b49-a85a-b05296aaa904",
                        ],
               ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
    <*errors.joinError | 0xc0015420c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*fmt.wrapError | 0xc000840400>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*fmt.wrapError | 0xc0008403e0>{
                    msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                    err: <*exported.ResponseError | 0xc0013a6450>{
                        ErrorCode: "LockContention",
                        StatusCode: 503,
                        RawResponse: {
                            Status: "503 Service Unavailable",
                            StatusCode: 503,
                            Proto: "HTTP/2.0",
                            ProtoMajor: 2,
                            ProtoMinor: 0,
                            Header: {
                                "Cache-Control": ["no-cache"],
                                "Content-Length": ["186"],
                                "X-Ms-Error-Code": ["LockContention"],
                                "X-Ms-Request-Id": [
                                    "57007d8e-6281-4f79-87e3-6b62771677b7",
                                ],
                                "X-Ms-Failure-Cause": ["service"],
                                "X-Ms-Ratelimit-Remaining-Subscription-Global-Deletes": ["2999"],
                                "X-Content-Type-Options": ["nosniff"],
                                "Date": [
                                    "Thu, 12 Mar 2026 20:29:07 GMT",
                                ],
                                "Pragma": ["no-cache"],
                                "Content-Type": ["application/json"],
                                "X-Envoy-Upstream-Service-Time": ["10009"],
                                "X-Ms-Operation-Identifier": [
                                    "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/9a19b057-0917-4393-a2cf-ff9874abdb4b",
                                ],
                                "X-Ms-Correlation-Request-Id": [
                                    "cd3510ea-1cec-45db-8f3f-4cac3c6e7c36",
                                ],
                                "Strict-Transport-Security": [
                                    "max-age=31536000; includeSubDomains",
                                ],
                                "X-Cache": ["CONFIG_NOCACHE"],
                                "X-Msedge-Ref": [
                                    "Ref A: DC15C07BB4844226852ED55513D0D6F1 Ref B: CHI261140507054 Re...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243]: Unexpected error:
    <*errors.joinError | 0xc0015420c0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 503: 503 Service Unavailable
    ERROR CODE: LockContention
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "LockContention",
        "message": "Failed to acquire lock: timed out",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*fmt.wrapError | 0xc000840400>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*fmt.wrapError | 0xc0008403e0>{
                    msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-pqj6m6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 503: 503 Service Unavailable\nERROR CODE: LockContention\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"LockContention\",\n    \"message\": \"Failed to acquire lock: timed out\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                    err: <*exported.ResponseError | 0xc0013a6450>{
                        ErrorCode: "LockContention",
                        StatusCode: 503,
                        RawResponse: {
                            Status: "503 Service Unavailable",
                            StatusCode: 503,
                            Proto: "HTTP/2.0",
                            ProtoMajor: 2,
                            ProtoMinor: 0,
                            Header: {
                                "Cache-Control": ["no-cache"],
                                "Content-Length": ["186"],
                                "X-Ms-Error-Code": ["LockContention"],
                                "X-Ms-Request-Id": [
                                    "57007d8e-6281-4f79-87e3-6b62771677b7",
                                ],
                                "X-Ms-Failure-Cause": ["service"],
                                "X-Ms-Ratelimit-Remaining-Subscription-Global-Deletes": ["2999"],
                                "X-Content-Type-Options": ["nosniff"],
                                "Date": [
                                    "Thu, 12 Mar 2026 20:29:07 GMT",
                                ],
                                "Pragma": ["no-cache"],
                                "Content-Type": ["application/json"],
                                "X-Envoy-Upstream-Service-Time": ["10009"],
                                "X-Ms-Operation-Identifier": [
                                    "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/9a19b057-0917-4393-a2cf-ff9874abdb4b",
                                ],
                                "X-Ms-Correlation-Request-Id": [
                                    "cd3510ea-1cec-45db-8f3f-4cac3c6e7c36",
                                ],
                                "Strict-Transport-Security": [
                                    "max-age=31536000; includeSubDomains",
                                ],
                                "X-Cache": ["CONFIG_NOCACHE"],
                                "X-Msedge-Ref": [
                                    "Ref A: DC15C07BB4844226852ED55513D0D6F1 Ref B: CHI261140507054 Re...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2032218705995239424
n/a2032169081666277376
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
full failure pattern:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e222.22%2 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 6: 0 · Mar 7: 0 · Mar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc001069d10>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
    {
        errs: [
            <*fmt.wrapError | 0xc00052cbc0>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                err: <*errors.errorString | 0xc0008a6470>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc001069d10>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
    {
        errs: [
            <*fmt.wrapError | 0xc00052cbc0>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                err: <*errors.errorString | 0xc0008a6470>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f034d0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
    {
        errs: [
            <*fmt.wrapError | 0xc00051c6c0>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                err: <*errors.errorString | 0xc00073e8e0>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: Timed out after 2400.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc000f034d0>: 
    VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial "4.20.15"
    {
        errs: [
            <*fmt.wrapError | 0xc00051c6c0>{
                msg: "VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.20.15) failed: no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                err: <*errors.errorString | 0xc00073e8e0>{
                    s: "no version in clusterversion/version status.history is greater than initial \"4.20.15\"",
                },
            },
        ],
    }
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelControl plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade using the candidate channel2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2032218705995239424
n/a2032203554982203392
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...
full failure pattern:
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e222.22%2 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 6: 0 · Mar 7: 0 · Mar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0015b3068>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-mc5p6j" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000a13260>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-mc5p6j\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000a13240>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-mc5p6j\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000a13200>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-mc5p6j\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb08a0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0015b3068>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-mc5p6j" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc000a13260>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-mc5p6j\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc000a13240>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-mc5p6j\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc000a13200>{
                        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-mc5p6j\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb08a0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012cc330>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="gpu-nodepool-cluster-7mb6hn" in resourcegroup="gpu-nodepools-NC4asT4v3-rspsmk" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00075dc00>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"gpu-nodepool-cluster-7mb6hn\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-rspsmk\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00075dbe0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"gpu-nodepool-cluster-7mb6hn\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-rspsmk\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00075dbc0>{
                        msg: "failed waiting for hcpCluster=\"gpu-nodepool-cluster-7mb6hn\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-rspsmk\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb06c0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:243] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012cc330>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="gpu-nodepool-cluster-7mb6hn" in resourcegroup="gpu-nodepools-NC4asT4v3-rspsmk" to finish deleting: context canceled
    {
        errs: [
            <*fmt.wrapError | 0xc00075dc00>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"gpu-nodepool-cluster-7mb6hn\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-rspsmk\" to finish deleting: context canceled",
                err: <*fmt.wrapError | 0xc00075dbe0>{
                    msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"gpu-nodepool-cluster-7mb6hn\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-rspsmk\" to finish deleting: context canceled",
                    err: <*fmt.wrapError | 0xc00075dbc0>{
                        msg: "failed waiting for hcpCluster=\"gpu-nodepool-cluster-7mb6hn\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-rspsmk\" to finish deleting: context canceled",
                        err: <*errors.errorString | 0xb06c0c0>{
                            s: "context canceled",
                        },
                    },
                },
            },
        ],
    }
occurred  
fail [:0]: A node timeout occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelHCP Nodepools GPU instances creates and deletes vm type NC4asT4v3 in a single cluster1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2032203554982203392
n/a2032126189862129664
ERROR CODE: InvalidRequestContent; detail message node pool <nodepool> '4.20.15' must not be greater than Control Plane version '4.20.8'; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message node pool <nodepool> '4.20.15' must not be greater than Control Plane version '4.20.8'; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 6: 0 · Mar 7: 0 · Mar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 9none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
    <*fmt.wrapError | 0xc00171c6c0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-g975c2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-g975c2\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00171c6a0>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-g975c2\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0009663c0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["170"],
                        "X-Ms-Correlation-Request-Id": [
                            "cf3a6b30-b9f9-40f5-860b-db7a4db99729",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260312T231200Z:cf3a6b30-b9f9-40f5-860b-db7a4db99729",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 1334633E7ABC4C9D961015632B4669DE Ref B: CHI261140502060 Ref C: 2026-03-12T23:11:59Z",
                        ],
                        "Date": [
                            "Thu, 12 Mar 2026 23:11:59 GMT",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["201"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/67d0f8c8-def2-4e27-b339-fb62d5a3d936",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "f425ba51-7086-4447-8016-171717a6c5ab",
                    ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:118]: Unexpected error:
    <*fmt.wrapError | 0xc00171c6c0>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "basic-hcp-cluster" in resourcegroup="basic-cluster-g975c2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-g975c2\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00171c6a0>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-g975c2\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/basic-cluster-g975c2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0009663c0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["170"],
                        "X-Ms-Correlation-Request-Id": [
                            "cf3a6b30-b9f9-40f5-860b-db7a4db99729",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260312T231200Z:cf3a6b30-b9f9-40f5-860b-db7a4db99729",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 1334633E7ABC4C9D961015632B4669DE Ref B: CHI261140502060 Ref C: 2026-03-12T23:11:59Z",
                        ],
                        "Date": [
                            "Thu, 12 Mar 2026 23:11:59 GMT",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["201"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/67d0f8c8-def2-4e27-b339-fb62d5a3d936",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Request-Id": [
                            "f425ba51-7086-4447-8016-171717a6c5ab",
                    ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:114]: Unexpected error:
    <*fmt.wrapError | 0xc001408120>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8t5md5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8t5md5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001408100>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8t5md5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000c90fc0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["2649"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260312T231012Z:014d9526-d846-4eb1-91c9-ca8a5c50bfbd",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 8080CB14631643F1BD10EDD83999FADD Ref B: CHI261140513054 Ref C: 2026-03-12T23:10:10Z",
                        ],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["170"],
                        "Expires": ["-1"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/9e990a7a-bc1c-4fa5-84e6-6d350f3c8cfe",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Thu, 12 Mar 2026 23:10:12 GMT",
                        ],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Ms-Correlation-Request-Id": [
                            "014d9526-d846-4eb1-91c9-ca8a5c50bfbd",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Ty...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:114]: Unexpected error:
    <*fmt.wrapError | 0xc001408120>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8t5md5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8t5md5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001408100>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8t5md5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8t5md5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000c90fc0>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["2649"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260312T231012Z:014d9526-d846-4eb1-91c9-ca8a5c50bfbd",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 8080CB14631643F1BD10EDD83999FADD Ref B: CHI261140513054 Ref C: 2026-03-12T23:10:10Z",
                        ],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["170"],
                        "Expires": ["-1"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/9e990a7a-bc1c-4fa5-84e6-6d350f3c8cfe",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Thu, 12 Mar 2026 23:10:12 GMT",
                        ],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Ms-Correlation-Request-Id": [
                            "014d9526-d846-4eb1-91c9-ca8a5c50bfbd",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Ty...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
    <*fmt.wrapError | 0xc00087e580>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-mq56rt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mq56rt\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00087e560>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mq56rt\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000e0e090>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "95277bb3-59de-4af9-a6be-d794c22ffea6",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: DECDB7E93FD643FFB79BD2D9D72D81DF Ref B: CHI261140510025 Ref C: 2026-03-12T22:52:58Z",
                        ],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["170"],
                        "Expires": ["-1"],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260312T225305Z:9b9678c6-264f-423d-bafc-7733fb8aa855",
                        ],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["7627"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/a50444fe-08f0-4f32-b20a-20c19b8c9e0e",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Correlation-Request-Id": [
                            "9b9678c6-264f-423d-bafc-7733fb8aa855",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
  ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:102]: Unexpected error:
    <*fmt.wrapError | 0xc00087e580>: 
    failed to create NodePool np-1: failed starting nodepool creation "np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-mq56rt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create NodePool np-1: failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mq56rt\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00087e560>{
            msg: "failed starting nodepool creation \"np-1\" for cluster \"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-mq56rt\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-mq56rt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster/nodePools/np-1\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidRequestContent\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidRequestContent\",\n    \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000e0e090>{
                ErrorCode: "InvalidRequestContent",
                StatusCode: 400,
                RawResponse: {
                    Status: "400 Bad Request",
                    StatusCode: 400,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "95277bb3-59de-4af9-a6be-d794c22ffea6",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: DECDB7E93FD643FFB79BD2D9D72D81DF Ref B: CHI261140510025 Ref C: 2026-03-12T22:52:58Z",
                        ],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["170"],
                        "Expires": ["-1"],
                        "X-Ms-Routing-Request-Id": [
                            "UKSOUTH:20260312T225305Z:9b9678c6-264f-423d-bafc-7733fb8aa855",
                        ],
                        "X-Ms-Error-Code": [
                            "InvalidRequestContent",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["7627"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/a50444fe-08f0-4f32-b20a-20c19b8c9e0e",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                        "X-Ms-Correlation-Request-Id": [
                            "9b9678c6-264f-423d-bafc-7733fb8aa855",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
  ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (9)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2032218705995239424
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
full failure pattern:
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 6: 0 · Mar 7: 0 · Mar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapErrors | 0xc00154e1e0>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-hb8wm7" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-hb8wm7\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001533930>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:170]: Unexpected error:
    <*fmt.wrapErrors | 0xc00154e1e0>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-hb8wm7" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-hb8wm7\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001533930>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-hb8wm7",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error:
    <*fmt.wrapErrors | 0xc001502510>: 
    failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-7rvs2t" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-7rvs2t\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000879850>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:103]: Unexpected error:
    <*fmt.wrapErrors | 0xc001502510>: 
    failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-7rvs2t" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-7rvs2t\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000879850>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster ea-cluster in resource group external-auth-cluster-7rvs2t",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2031998313082916864
ERROR CODE: DeploymentFailed; detail code InvalidRequestContent; detail message node pool <nodepool> '4.20.15' must not be greater than Control Plane version '4.20.8'
full failure pattern:
ERROR CODE: DeploymentFailed; detail code InvalidRequestContent; detail message node pool <nodepool> '4.20.15' must not be greater than Control Plane version '4.20.8'
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 6: 0 · Mar 7: 0 · Mar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:100]: Unexpected error:
    <*fmt.wrapError | 0xc000fc6740>: 
    failed waiting for deployment "aro-hcp-demo" in resourcegroup="gpu-nodepools-NC4asT4v3-w6hmz9" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/aro-hcp-demo/operationStatuses/08584282515392752534
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/nodepool-1\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidRequestContent\",\r\n            \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed waiting for deployment \"aro-hcp-demo\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-w6hmz9\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/aro-hcp-demo/operationStatuses/08584282515392752534\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/nodepool-1\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"InvalidRequestContent\\\",\\r\\n            \\\"message\\\": \\\"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001500e10>{
            ErrorCode: "DeploymentFailed",
            StatusCode: 200,
            RawResponse: {
                Status: "200 OK",
                StatusCode: 200,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Pragma": ["no-cache"],
                    "Content-Length": ["1243"],
                    "X-Ms-Request-Id": [
                        "0d231ad8-8318-4e3a-920c-9541e2affd70",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Correlation-Request-Id": [
                        "0d231ad8-8318-4e3a-920c-9541e2affd70",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Content-Type": [
                        "application/json; charset=utf-8",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260312T225347Z:0d231ad8-8318-4e3a-920c-9541e2affd70",
                    ],
                    "Expires": ["-1"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 4733BC488DE8424898A45EF16542E708 Ref B: CH1AA2020603045 Ref C: 2026-03-12T22:53:47Z",
                    ],
                    "Date": [
                        "Thu, 12 Mar 2026 22:53:46 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000fc6620>{
                    s:...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:100]: Unexpected error:
    <*fmt.wrapError | 0xc000fc6740>: 
    failed waiting for deployment "aro-hcp-demo" in resourcegroup="gpu-nodepools-NC4asT4v3-w6hmz9" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/aro-hcp-demo/operationStatuses/08584282515392752534
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/nodepool-1\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidRequestContent\",\r\n            \"message\": \"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed waiting for deployment \"aro-hcp-demo\" in resourcegroup=\"gpu-nodepools-NC4asT4v3-w6hmz9\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/aro-hcp-demo/operationStatuses/08584282515392752534\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/gpu-nodepools-NC4asT4v3-w6hmz9/providers/Microsoft.Resources/deployments/nodepool-1\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"InvalidRequestContent\\\",\\r\\n            \\\"message\\\": \\\"Node pool version '4.20.15' must not be greater than Control Plane version '4.20.8'\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001500e10>{
            ErrorCode: "DeploymentFailed",
            StatusCode: 200,
            RawResponse: {
                Status: "200 OK",
                StatusCode: 200,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Pragma": ["no-cache"],
                    "Content-Length": ["1243"],
                    "X-Ms-Request-Id": [
                        "0d231ad8-8318-4e3a-920c-9541e2affd70",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Correlation-Request-Id": [
                        "0d231ad8-8318-4e3a-920c-9541e2affd70",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Content-Type": [
                        "application/json; charset=utf-8",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260312T225347Z:0d231ad8-8318-4e3a-920c-9541e2affd70",
                    ],
                    "Expires": ["-1"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 4733BC488DE8424898A45EF16542E708 Ref B: CH1AA2020603045 Ref C: 2026-03-12T22:53:47Z",
                    ],
                    "Date": [
                        "Thu, 12 Mar 2026 22:53:46 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000fc6620>{
                    s:...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelHCP Nodepools GPU instances creates and deletes vm type NC4asT4v3 in a single cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2032218705995239424
Should receive valid HTTP status code from authorized IP
full failure pattern:
Should receive valid HTTP status code from authorized IP
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 6: 0 · Mar 7: 0 · Mar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:168]: Timed out after 120.002s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:167 with:
Should receive valid HTTP status code from authorized IP
Expected
    <string>: 000
to match regular expression
    <string>: ^[2-5][0-9][0-9]$
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:168]: Timed out after 120.002s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:167 with:
Should receive valid HTTP status code from authorized IP
Expected
    <string>: 000
to match regular expression
    <string>: ^[2-5][0-9][0-9]$
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2032084631825682432
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsMar 6: 0 · Mar 7: 0 · Mar 8: 0 · Mar 9: 0 · Mar 10: 0 · Mar 11: 0 · Mar 12: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapErrors | 0xc00032e8d0>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-zkjhtw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-zkjhtw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0014f4be0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw",
            },
            <*fmt.wrapErrors | 0xc00032e8a0>{
                msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-zkjhtw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0014f4be0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapErrors | 0xc00032e8d0>: 
    failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-zkjhtw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster node-limit-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-zkjhtw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0014f4be0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw",
            },
            <*fmt.wrapErrors | 0xc00032e8a0>{
                msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-zkjhtw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0014f4be0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-zkjhtw",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2031998313082916864