CIHealth
<
Single Day: Apr 3▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: STG

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 14
e2e (14)
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift
02:00:44 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:164]: Unexpected error:
    <*fmt.wrapError | 0xc000786700>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcr7xr-w9cl9v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcr7xr/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcr7xr-w9cl9v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcr7xr/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000984f90>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260403T024529Z:507906a9-3741-4c35-99cf-b4f6b8fe9a4b",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 4BC4BC7D598E4A38B774049176C0C5B8 Ref B: CHI261140508025 Ref C: 2026-04-03T02:45:28Z",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/75948d99-9fdd-49dc-a774-686a518c88f9",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Fri, 03 Apr 2026 02:45:28 GMT",
                    ],
                    "Content-Length": ["107"],
                    "X-Ms-Request-Id": [
                        "e9eebcd1-0444-496b-90c5-43c9c0a830e0",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["247"],
                    "X-Ms-Correlation-Request-Id": [
                        "507906a9-3741-4c35-99cf-b4f6b8fe9a4b",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Pragma": ["no-cache"],
                    "Content-Type": ["application/json"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000786640>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:164]: Unexpected error:
    <*fmt.wrapError | 0xc000786700>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcr7xr-w9cl9v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcr7xr/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcr7xr-w9cl9v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcr7xr/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000984f90>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260403T024529Z:507906a9-3741-4c35-99cf-b4f6b8fe9a4b",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 4BC4BC7D598E4A38B774049176C0C5B8 Ref B: CHI261140508025 Ref C: 2026-04-03T02:45:28Z",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/75948d99-9fdd-49dc-a774-686a518c88f9",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Fri, 03 Apr 2026 02:45:28 GMT",
                    ],
                    "Content-Length": ["107"],
                    "X-Ms-Request-Id": [
                        "e9eebcd1-0444-496b-90c5-43c9c0a830e0",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["247"],
                    "X-Ms-Correlation-Request-Id": [
                        "507906a9-3741-4c35-99cf-b4f6b8fe9a4b",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Pragma": ["no-cache"],
                    "Content-Type": ["application/json"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000786640>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f383c0>: 
    failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-cptczp" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-cptczp, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-cptczp\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-cptczp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012e8120>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-cptczp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:185]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f383c0>: 
    failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-cptczp" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-cptczp, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-cptczp\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-cptczp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012e8120>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-cptczp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc001608600>: 
    failed waiting for hcpCluster="cluster-zstream-4-20-c42d4w" in resourcegroup="rg-zstream-upgrade-4-20-dp7xfn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-c42d4w in resource group rg-zstream-upgrade-4-20-dp7xfn, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-c42d4w\" in resourcegroup=\"rg-zstream-upgrade-4-20-dp7xfn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-c42d4w in resource group rg-zstream-upgrade-4-20-dp7xfn, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009bc6e0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-c42d4w in resource group rg-zstream-upgrade-4-20-dp7xfn",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:151]: Unexpected error:
    <*fmt.wrapErrors | 0xc001608600>: 
    failed waiting for hcpCluster="cluster-zstream-4-20-c42d4w" in resourcegroup="rg-zstream-upgrade-4-20-dp7xfn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-c42d4w in resource group rg-zstream-upgrade-4-20-dp7xfn, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-zstream-4-20-c42d4w\" in resourcegroup=\"rg-zstream-upgrade-4-20-dp7xfn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-c42d4w in resource group rg-zstream-upgrade-4-20-dp7xfn, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009bc6e0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-zstream-4-20-c42d4w in resource group rg-zstream-upgrade-4-20-dp7xfn",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP stable channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error:
    <*fmt.wrapErrors | 0xc00095d9b0>: 
    failed waiting for hcpCluster="cluster-stable-4-20-f9n864" in resourcegroup="rg-stable-4-20-2p52mh" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-f9n864 in resource group rg-stable-4-20-2p52mh, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-f9n864\" in resourcegroup=\"rg-stable-4-20-2p52mh\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-f9n864 in resource group rg-stable-4-20-2p52mh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00097a8d0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-f9n864 in resource group rg-stable-4-20-2p52mh",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:125]: Unexpected error:
    <*fmt.wrapErrors | 0xc00095d9b0>: 
    failed waiting for hcpCluster="cluster-stable-4-20-f9n864" in resourcegroup="rg-stable-4-20-2p52mh" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-f9n864 in resource group rg-stable-4-20-2p52mh, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-stable-4-20-f9n864\" in resourcegroup=\"rg-stable-4-20-2p52mh\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-f9n864 in resource group rg-stable-4-20-2p52mh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00097a8d0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-stable-4-20-f9n864 in resource group rg-stable-4-20-2p52mh",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010c28d0>: 
    failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-cl9lw2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-cl9lw2, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-cl9lw2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-cl9lw2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010a0280>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-cl9lw2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010c28d0>: 
    failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-cl9lw2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-cl9lw2, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-cl9lw2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-cl9lw2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010a0280>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-cl9lw2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:127]: Unexpected error:
    <*fmt.wrapErrors | 0xc001158a80>: 
    failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5sbjzh" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-5sbjzh, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5sbjzh\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-5sbjzh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008253a0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-5sbjzh",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:127]: Unexpected error:
    <*fmt.wrapErrors | 0xc001158a80>: 
    failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5sbjzh" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-5sbjzh, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5sbjzh\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-5sbjzh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008253a0>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster negative-tests-cluster in resource group rg-negative-tests-5sbjzh",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift
02:00:44 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:165]: Unexpected error:
    <*fmt.wrapError | 0xc000a8eb60>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5cvprg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5cvprg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000ed5a40>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Correlation-Request-Id": [
                        "d7fc2f57-2287-44c2-b700-be1aaa0497cb",
                    ],
                    "Date": [
                        "Fri, 03 Apr 2026 02:45:24 GMT",
                    ],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["220"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Content-Type-Options": ["nosniff"],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260403T024524Z:d7fc2f57-2287-44c2-b700-be1aaa0497cb",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Expires": ["-1"],
                    "X-Ms-Request-Id": [
                        "0c717bf9-7d9a-4776-9e41-ba423e577bf9",
                    ],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/00474633-06a8-48ff-b174-93216b11a56c",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: 1B9855FF492941F7BAEF61CDF010807E Ref B: CHI261140510054 Ref C: 2026-04-03T02:45:24Z",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000a8ea80>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "mana...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:165]: Unexpected error:
    <*fmt.wrapError | 0xc000a8eb60>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5cvprg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-5cvprg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000ed5a40>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Correlation-Request-Id": [
                        "d7fc2f57-2287-44c2-b700-be1aaa0497cb",
                    ],
                    "Date": [
                        "Fri, 03 Apr 2026 02:45:24 GMT",
                    ],
                    "Content-Type": ["application/json"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["220"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Content-Type-Options": ["nosniff"],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260403T024524Z:d7fc2f57-2287-44c2-b700-be1aaa0497cb",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Expires": ["-1"],
                    "X-Ms-Request-Id": [
                        "0c717bf9-7d9a-4776-9e41-ba423e577bf9",
                    ],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/00474633-06a8-48ff-b174-93216b11a56c",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: 1B9855FF492941F7BAEF61CDF010807E Ref B: CHI261140510054 Ref C: 2026-04-03T02:45:24Z",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000a8ea80>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "mana...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:245]: Unexpected error:
    <*fmt.wrapErrors | 0xc000fe0f60>: 
    failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-d4j72k" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-d4j72k, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-d4j72k\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-d4j72k, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a19280>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-d4j72k",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:245]: Unexpected error:
    <*fmt.wrapErrors | 0xc000fe0f60>: 
    failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-d4j72k" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-d4j72k, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-d4j72k\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-d4j72k, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a19280>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster oidc-wi-cluster in resource group oidc-wi-d4j72k",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc0019716e0>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-xm9l27" in resourcegroup="admin-credential-lifecycle-test-m8sr7m" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-xm9l27 in resource group admin-credential-lifecycle-test-m8sr7m, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-xm9l27\" in resourcegroup=\"admin-credential-lifecycle-test-m8sr7m\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-xm9l27 in resource group admin-credential-lifecycle-test-m8sr7m, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010d6b90>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-xm9l27 in resource group admin-credential-lifecycle-test-m8sr7m",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:174]: Unexpected error:
    <*fmt.wrapErrors | 0xc0019716e0>: 
    failed waiting for hcpCluster="admin-cred-lifecycle-xm9l27" in resourcegroup="admin-credential-lifecycle-test-m8sr7m" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-xm9l27 in resource group admin-credential-lifecycle-test-m8sr7m, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"admin-cred-lifecycle-xm9l27\" in resourcegroup=\"admin-credential-lifecycle-test-m8sr7m\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-xm9l27 in resource group admin-credential-lifecycle-test-m8sr7m, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0010d6b90>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster admin-cred-lifecycle-xm9l27 in resource group admin-credential-lifecycle-test-m8sr7m",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:164]: Unexpected error:
    <*fmt.wrapErrors | 0xc000926600>: 
    failed waiting for hcpCluster="np-version-upgrade-cluster-lrm9b9" in resourcegroup="rg-np-version-upgrade-lrm9b9-j2xf2m" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-lrm9b9 in resource group rg-np-version-upgrade-lrm9b9-j2xf2m, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"np-version-upgrade-cluster-lrm9b9\" in resourcegroup=\"rg-np-version-upgrade-lrm9b9-j2xf2m\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-lrm9b9 in resource group rg-np-version-upgrade-lrm9b9-j2xf2m, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0000d3770>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-lrm9b9 in resource group rg-np-version-upgrade-lrm9b9-j2xf2m",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:164]: Unexpected error:
    <*fmt.wrapErrors | 0xc000926600>: 
    failed waiting for hcpCluster="np-version-upgrade-cluster-lrm9b9" in resourcegroup="rg-np-version-upgrade-lrm9b9-j2xf2m" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-lrm9b9 in resource group rg-np-version-upgrade-lrm9b9-j2xf2m, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"np-version-upgrade-cluster-lrm9b9\" in resourcegroup=\"rg-np-version-upgrade-lrm9b9-j2xf2m\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-lrm9b9 in resource group rg-np-version-upgrade-lrm9b9-j2xf2m, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0000d3770>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-lrm9b9 in resource group rg-np-version-upgrade-lrm9b9-j2xf2m",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error:
    <*fmt.wrapErrors | 0xc00163a780>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-vf9wp2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-vf9wp2, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-vf9wp2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-vf9wp2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000342170>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-vf9wp2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:132]: Unexpected error:
    <*fmt.wrapErrors | 0xc00163a780>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-vf9wp2" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-vf9wp2, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-vf9wp2\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-vf9wp2, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000342170>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-vf9wp2",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f14ae0>: 
    failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-7kzsrx" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-7kzsrx, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-7kzsrx\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-7kzsrx, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000792f60>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-7kzsrx",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cilium.go:126]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f14ae0>: 
    failed waiting for hcpCluster="cilium-cluster" in resourcegroup="e2e-cilium-7kzsrx" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-7kzsrx, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cilium-cluster\" in resourcegroup=\"e2e-cilium-7kzsrx\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-7kzsrx, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000792f60>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cluster in resource group e2e-cilium-7kzsrx",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift
02:00:44 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error:
    <*exported.ResponseError | 0xc000f12ff0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-kvgxbs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InternalServerError",
        StatusCode: 500,
        RawResponse: {
            Status: "500 Internal Server Error",
            StatusCode: 500,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Expires": ["-1"],
                "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                "X-Cache": ["CONFIG_NOCACHE"],
                "Pragma": ["no-cache"],
                "Content-Length": ["107"],
                "Content-Type": ["application/json"],
                "X-Msedge-Ref": [
                    "Ref A: 8D16791A614747E68D39E114ED8D2C9B Ref B: CHI261140501034 Ref C: 2026-04-03T02:45:28Z",
                ],
                "Cache-Control": ["no-cache"],
                "X-Ms-Request-Id": [
                    "a740f347-b039-46b8-854b-bf971aa079a8",
                ],
                "X-Ms-Failure-Cause": ["service"],
                "X-Ms-Operation-Identifier": [
                    "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/1c1f941e-ebbc-4aa6-b2fa-002644ef4aff",
                ],
                "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                "X-Ms-Correlation-Request-Id": [
                    "1951c97d-99ef-42e9-90a4-b7c5a6757c6d",
                ],
                "X-Content-Type-Options": ["nosniff"],
                "Date": [
                    "Fri, 03 Apr 2026 02:45:29 GMT",
                ],
                "X-Ms-Error-Code": [
                    "InternalServerError",
                ],
                "X-Envoy-Upstream-Service-Time": ["508"],
                "X-Ms-Routing-Request-Id": [
                    "UKSOUTH:20260403T024529Z:1951c97d-99ef-42e9-90a4-b7c5a6757c6d",
                ],
                "Strict-Transport-Security": [
                    "max-age=31536000; includeSubDomains",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000e149a0>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 107,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "management.azure.com",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-kvgxbs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2024-06-10-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
                    ],
                    "Authorization": ["redacted"],
                    "Accept": ["application/json"],
                    "Content-Length": ["50"],
                    "Content-Type": ["application/json"],
    ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error:
    <*exported.ResponseError | 0xc000f12ff0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-kvgxbs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        ErrorCode: "InternalServerError",
        StatusCode: 500,
        RawResponse: {
            Status: "500 Internal Server Error",
            StatusCode: 500,
            Proto: "HTTP/2.0",
            ProtoMajor: 2,
            ProtoMinor: 0,
            Header: {
                "Expires": ["-1"],
                "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                "X-Cache": ["CONFIG_NOCACHE"],
                "Pragma": ["no-cache"],
                "Content-Length": ["107"],
                "Content-Type": ["application/json"],
                "X-Msedge-Ref": [
                    "Ref A: 8D16791A614747E68D39E114ED8D2C9B Ref B: CHI261140501034 Ref C: 2026-04-03T02:45:28Z",
                ],
                "Cache-Control": ["no-cache"],
                "X-Ms-Request-Id": [
                    "a740f347-b039-46b8-854b-bf971aa079a8",
                ],
                "X-Ms-Failure-Cause": ["service"],
                "X-Ms-Operation-Identifier": [
                    "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/uksouth/1c1f941e-ebbc-4aa6-b2fa-002644ef4aff",
                ],
                "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                "X-Ms-Correlation-Request-Id": [
                    "1951c97d-99ef-42e9-90a4-b7c5a6757c6d",
                ],
                "X-Content-Type-Options": ["nosniff"],
                "Date": [
                    "Fri, 03 Apr 2026 02:45:29 GMT",
                ],
                "X-Ms-Error-Code": [
                    "InternalServerError",
                ],
                "X-Envoy-Upstream-Service-Time": ["508"],
                "X-Ms-Routing-Request-Id": [
                    "UKSOUTH:20260403T024529Z:1951c97d-99ef-42e9-90a4-b7c5a6757c6d",
                ],
                "Strict-Transport-Security": [
                    "max-age=31536000; includeSubDomains",
                ],
            },
            Body: <*exported.nopClosingBytesReader | 0xc000e149a0>{
                s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                i: 0,
            },
            ContentLength: 107,
            TransferEncoding: nil,
            Close: false,
            Uncompressed: false,
            Trailer: nil,
            Request: {
                Method: "PATCH",
                URL: {
                    Scheme: "https",
                    Opaque: "",
                    User: nil,
                    Host: "management.azure.com",
                    Path: "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-kvgxbs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster",
                    RawPath: "",
                    OmitHost: false,
                    ForceQuery: false,
                    RawQuery: "api-version=2024-06-10-preview",
                    Fragment: "",
                    RawFragment: "",
                },
                Proto: "HTTP/1.1",
                ProtoMajor: 1,
                ProtoMinor: 1,
                Header: {
                    "User-Agent": [
                        "azsdk-go-armredhatopenshifthcp/v0.1.0 (go1.25.7 (Red Hat 1.25.7-1.el9_7) X:strictfipsruntime; linux)",
                    ],
                    "Authorization": ["redacted"],
                    "Accept": ["application/json"],
                    "Content-Length": ["50"],
                    "Content-Type": ["application/json"],
    ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
02:00:44 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ac6f60>: 
    failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-xvmm2h" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xvmm2h, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-xvmm2h\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xvmm2h, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001381f40>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xvmm2h",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:105]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ac6f60>: 
    failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-xvmm2h" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xvmm2h, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-xvmm2h\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xvmm2h, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001381f40>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster disabled-image-registry-hcp-cluster in resource group disabled-image-registry-xvmm2h",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred