CIHealth
<
Single Day: Mar 12▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 07bee9f · merge b1ab516
 n/a n/a passed #4432 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 8a1ae2a · merge 8a1ae2a
 n/a e2e failed ★#4399 (merged) 1
e2e (1)
Yet, Eventually failed because the matcher was not satisfied:
22:43:12 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.250s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
    <v1.PodPhase>: Pending
to equal
    <v1.PodPhase>: Succeeded
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.250s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
    <v1.PodPhase>: Pending
to equal
    <v1.PodPhase>: Succeeded
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head f9b1e39 · merge ff3b139
 n/a n/a passed #4429 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head dcec1a0
 n/a n/a passed #4393 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 0533aaf · merge 855eea8
 n/a unknown failed #4425 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 2cc3d47 · merge 2cc3d47
 n/a n/a passed #4390 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 2cc3d47 · merge 2cc3d47
 n/a e2e failed ★#4390 (merged) 1
e2e (1)
Yet, Eventually failed because the matcher was not satisfied:
20:52:55 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.254s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
    <v1.PodPhase>: Running
to equal
    <v1.PodPhase>: Succeeded
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.254s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
    <v1.PodPhase>: Running
to equal
    <v1.PodPhase>: Succeeded
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 9128c3e · merge 9128c3e
 n/a e2e failed ★#4383 (merged) 1
e2e (1)
Yet, Eventually failed because the matcher was not satisfied:
20:50:10 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.255s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
    <v1.PodPhase>: Pending
to equal
    <v1.PodPhase>: Succeeded
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:314]: Told to stop trying after 20.255s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
    <v1.PodPhase>: Pending
to equal
    <v1.PodPhase>: Succeeded
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0225b04 · merge 0225b04
 n/a n/a passed #4434 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 37ef6be · merge 37ef6be
 n/a n/a passed #4437 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 45a701a · merge 45a701a
 n/a n/a passed #4438 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 126c161 · merge 126c161
 n/a n/a passed #4436 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head c8926b8 · merge 8550ece
 n/a n/a passed #4219 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 3787d3d
 n/a n/a passed #4281 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 9128c3e · merge 9128c3e
 n/a n/a passed #4383 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 2cc3d47 · merge 2cc3d47
 n/a n/a passed #4390 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 6e42c17
 n/a n/a passed #4431 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 2cc3d47 · merge 2cc3d47
 n/a n/a passed #4390 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head f76ac24 · merge 76945bf
 n/a provision failed ⚠#4384 (merged) 1
provision (1)
failed post-install: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress context deadline exceeded
18:12:13 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ACM/management/deploy-mce-config · suite step graph
Show raw failure
time=2026-03-12T19:05:37.116Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config description="Step deploy-mce-config\n Kind: Helm\n"
time=2026-03-12T19:05:37.680Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-03-12T19:05:37.680Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-12T19:05:37.741Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config namespace=multicluster-engine
time=2026-03-12T19:05:37.855Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config dryRun=false
time=2026-03-12T19:05:37.855Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-12T19:05:37.966Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-12T19:11:04.663Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="failed post-install: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded"
time=2026-03-12T19:11:04.663Z level=INFO msg="Finished deploying Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-12T19:11:04.663Z level=INFO msg="Running inline diagnostics." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-12T19:11:10.563Z level=INFO msg="Determined release status." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config release=mce-config namespace=multicluster-engine status=failed description="Release \"mce-config\" failed: failed post-install: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded" values="map[finalizeMceJobImage:mcr.microsoft.com/aks/command/runtime@sha256:1c49cb26c44ce9f846257ca1bae2d275862ebeb31d5529a62a6c4075665fd38f global:map[registryOverride:arohcpsvcdev.azurecr.io/acm-d-cache] localCluster:map[addonConfig:map[applicationManagerEnabled:false certPolicyControllerEnabled:false policyControllerEnabled:true searchCollectorEnabled:false] kubeApiUrl:https://kubernetes.default.svc] policy:map[cluster-lifecycle:map[hubconfig:map[replicaCount:2]] grc:map[hubconfig:map[replicaCount:2]]]]"
time=2026-03-12T19:11:10.565Z level=INFO msg="Kube-events kusto link for troubleshooting:" serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config url="https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs?query=H4sIAAAAAAAA/6yVT4/aPBDG7/kUo5wWKVkB+76tmmoPdNvTqq3U7WmrHhx7CF6cmcg2ICQ+fGWSkJQ/hUV74PIwz8+e8UwmMujBouOFlejgHpTwwovc4A3nLyj9oyaVOW81FQnU0jdR4k7CJdJWcZWQrTz4FQEAxA9m4TzaH2wwTiDmCimVtZaWgkSBJZLP5rVk0KdCKaZUMnnLxqBNl+PgjJNXAQsrs0a2TegVhIqNluvmRsdpLaawMg2/2tJk0QaXC+N1ewZSoQlP+S+0fNKkNBX/PDnvYl5PuxzwGSvD61CzEHj+JS9GHSTVoa7lVJYrUQjP5wiPuzQm4egHpqneFsKwFKZthiPCvr1zFoZzYfq984R2qSVOpOTFG1TvEHdQQSeuYpy19Waq10tvOu0XcPeGvte+1/NOfAKOsCdKfaeu87qHr80zdh5VSLWRz5fz6+4qW3Yw1fb2Uf8eiPPugpdoSZDEljC1osQV23kfUDtVw3nCbR/M1hVaN9NT3ytRWxK3tUe/P0bzRY6W0KP7EvaCizawmqFF8LpE50VZQY5+hUhwo4THIN/E4+H4XTq8S0fjn6MP2fD/7O79czyA21s4FTMaZcP/nuPBINrAC2uCuSZ1r4nQ9rYZE3RLrL+99tdWtIHKcvi3u2gCFas6+DJIAoaLBEp0ThQByVahhXzdS16hk38CAAD//0DSs6VzBwAA"
time=2026-03-12T19:11:10.567Z level=INFO msg="Deployment complete." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config
time=2026-03-12T19:11:10.568Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ACM resourceGroup=management step=deploy-mce-config err="error running Helm release deployment Step, failed to deploy helm release: failed post-install: resource not ready, name: finalize-mce-config, kind: Job, status: InProgress\ncontext deadline exceeded"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head f76ac24 · merge 76945bf
 n/a n/a passed #4384 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 0b33790 · merge 0b33790
 n/a n/a passed #4376 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 8a1ae2a · merge 8a1ae2a
 n/a n/a passed #4399 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head dcbf64b · merge 61fd736
 n/a n/a passed #4409 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head c0fed25
 n/a n/a passed #4305 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 599dd2c · merge 599dd2c
 n/a n/a passed #4433 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 9128c3e · merge 9128c3e
 n/a n/a passed #4383 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head f76ac24 · merge 76945bf
 n/a n/a passed #4384 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 1fcabe1
 n/a unknown failed #4435 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0b33790 · merge 0b33790
 n/a n/a passed #4376 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 0b33790 · merge 0b33790
 n/a n/a passed #4376 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head d1b9263
 n/a unknown failed #4305 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 0b33790 · merge 0b33790
 n/a n/a passed #4376 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head aaefafa · merge 0225b04
 n/a e2e failed #4434 (merged) 1
e2e (1)
timeout during CreateHCPClusterAndWait; context deadline exceeded
15:23:50 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ec3830>: 
    failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-4l5s9w" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w, error: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-4l5s9w\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fd5a40>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w",
            },
            <*fmt.wrapErrors | 0xc000ec3800>{
                msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-4l5s9w\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fd5a40>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ec3830>: 
    failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w, error: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-4l5s9w" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster negative-tests-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w, error: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-4l5s9w\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fd5a40>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w",
            },
            <*fmt.wrapErrors | 0xc000ec3800>{
                msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-4l5s9w\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fd5a40>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster negative-tests-cluster in resource group rg-negative-tests-4l5s9w",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head ebe6847
 n/a unknown failed #4435 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 3c72a8d · merge 61fd736
 n/a e2e failed ★#4409 (merged) 1
e2e (1)
Yet, Eventually failed because the matcher was not satisfied:
14:47:06 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:310]: Told to stop trying after 20.283s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
    <v1.PodPhase>: Pending
to equal
    <v1.PodPhase>: Succeeded
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:310]: Told to stop trying after 20.283s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
    <v1.PodPhase>: Pending
to equal
    <v1.PodPhase>: Succeeded
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 8ec6255 · merge 599dd2c
 n/a unknown failed #4433 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 0b33790 · merge 0b33790
 n/a n/a passed #4376 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head dca081b · merge a82c625
 n/a n/a passed #4347 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 9128c3e · merge 9128c3e
 n/a n/a passed #4383 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head edbec11 · merge 80403bb
 n/a n/a passed #4407 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 0cc56c0 · merge b1ab516
 n/a e2e failed ⚠#4432 (merged) 1
e2e (1)
ERROR CODE: InternalServerError; detail message failed to create transaction: pq: sorry, too many clients already; provider Microsoft.RedHatOpenShift
12:52:08 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapError | 0xc00050cd20>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-pzvdp4" to finish creating: GET https://rp.j6093568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599",
      "name": "a9704dbd-416f-4146-ab33-41bba4f19599",
      "status": "Failed",
      "startTime": "2026-03-12T14:02:34.543107268Z",
      "endTime": "2026-03-12T14:02:58.465199148Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pzvdp4\" to finish creating: GET https://rp.j6093568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599\",\n  \"name\": \"a9704dbd-416f-4146-ab33-41bba4f19599\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-12T14:02:34.543107268Z\",\n  \"endTime\": \"2026-03-12T14:02:58.465199148Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to create transaction: pq: sorry, too many clients already\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000883f60>{
            msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pzvdp4\" to finish creating: GET https://rp.j6093568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599\",\n  \"name\": \"a9704dbd-416f-4146-ab33-41bba4f19599\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-12T14:02:34.543107268Z\",\n  \"endTime\": \"2026-03-12T14:02:58.465199148Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to create transaction: pq: sorry, too many clients already\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000e4cc00>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "353143a4-e740-4e0f-8803-d05dbb5d0c04",
                        ],
                        "Date": [
                            "Thu, 12 Mar 2026 14:03:04 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000883e60>{
                        s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapError | 0xc00050cd20>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-pzvdp4" to finish creating: GET https://rp.j6093568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599",
      "name": "a9704dbd-416f-4146-ab33-41bba4f19599",
      "status": "Failed",
      "startTime": "2026-03-12T14:02:34.543107268Z",
      "endTime": "2026-03-12T14:02:58.465199148Z",
      "error": {
        "code": "InternalServerError",
        "message": "failed to create transaction: pq: sorry, too many clients already"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pzvdp4\" to finish creating: GET https://rp.j6093568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599\",\n  \"name\": \"a9704dbd-416f-4146-ab33-41bba4f19599\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-12T14:02:34.543107268Z\",\n  \"endTime\": \"2026-03-12T14:02:58.465199148Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to create transaction: pq: sorry, too many clients already\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000883f60>{
            msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-pzvdp4\" to finish creating: GET https://rp.j6093568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9704dbd-416f-4146-ab33-41bba4f19599\",\n  \"name\": \"a9704dbd-416f-4146-ab33-41bba4f19599\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-12T14:02:34.543107268Z\",\n  \"endTime\": \"2026-03-12T14:02:58.465199148Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"failed to create transaction: pq: sorry, too many clients already\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000e4cc00>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "Server": ["istio-envoy"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "353143a4-e740-4e0f-8803-d05dbb5d0c04",
                        ],
                        "Date": [
                            "Thu, 12 Mar 2026 14:03:04 GMT",
                        ],
                    },
                    Body: <*exported.nopClosingBytesReader | 0xc000883e60>{
                        s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 57, 55, 52, 101, 98, 100, 52, 54, 45, 56, 97, 100, 51, 45, 52, 49, 101, 51, 45, 97, 102, 101, 102, 45, 55, 101, 102, 50, 53, 102, 100, 53, 99, 51, 55, 49, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 119, 101, 115, 116, 117, 115, 51, 47, 104, 99, 112, 79,...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 96010e5 · merge ff3b139
 n/a n/a passed #4429 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 5b9e8e9 · merge 5b9e8e9
 n/a n/a passed #4416 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 51bcb1f
 n/a n/a passed #4431 (closed) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head dacf876
 n/a unknown failed #4391 (open) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0b33790 · merge 0b33790
 n/a n/a passed #4376 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 147684b
 n/a unknown failed #4419 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head ee63326 · merge ee63326
 n/a n/a passed #4428 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 9128c3e · merge 9128c3e
 n/a n/a passed #4383 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9e1e02c
 n/a n/a passed #4281 (open) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 6dbdab4 · merge ee63326
 n/a provision failed ⚠#4428 (merged) 1
provision (1)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError("HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve
10:30:21 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.RouteMonitorOperator/global/mirror-blackbox-image · suite step graph
Show raw failure
time=2026-03-12T11:23:21.926Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=global step=mirror-blackbox-image description="Step mirror-blackbox-image\n  Kind: ImageMirror\n  From {<nil> routeMonitorOperator.blackboxExporterImage.registry <nil>}:{<nil> routeMonitorOperator.blackboxExporterImage.repository <nil>}@{<nil> routeMonitorOperator.blackboxExporterImage.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T11:23:55.146Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RouteMonitorOperator resourceGroup=global step=mirror-blackbox-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org registry.build02.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nFailed to log in to ACR arohcpsvcdev: ERROR: The command failed with an unexpected error. Here is the traceback:\nERROR: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 204, in _new_conn\n    sock = connection.create_connection(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/connection.py\", line 60, in create_connection\n    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/python3.12/socket.py\", line 963, in getaddrinfo\n    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nsocket.gaierror: [Errno -2] Name or service not known\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 787, in urlopen\n    response = self._make_request(\n               ^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 488, in _make_request\n    raise new_e\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 464, in _make_request\n    self._validate_conn(conn)\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 1093, in _validate_conn\n    conn.connect()\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 759, in connect\n    self.sock = sock = self._new_conn()\n                       ^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connection.py\", line 211, in _new_conn\n    raise NameResolutionError(self.host, self, e) from e\nurllib3.exceptions.NameResolutionError: HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 667, in send\n    resp = conn.urlopen(\n           ^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/connectionpool.py\", line 841, in urlopen\n    retries = retries.increment(\n              ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/urllib3/util/retry.py\", line 535, in increment\n    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nurllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/usr/lib64/az/lib/python3.12/site-packages/knack/cli.py\", line 233, in invoke\n    cmd_result = self.invocation.execute(args)\n                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 682, in execute\n    raise ex\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 812, in _run_jobs_serially\n    results.append(self._run_job(expanded_arg, cmd_copy))\n                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 781, in _run_job\n    result = cmd_copy(params)\n             ^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/__init__.py\", line 336, in __call__\n    return self.handler(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/core/commands/command_operation.py\", line 120, in handler\n    return op(**command_args)\n           ^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/custom.py\", line 294, in acr_login\n    login_server, _, password = get_login_credentials(\n                                ^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 505, in get_login_credentials\n    return _get_credentials(cmd,\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 449, in _get_credentials\n    return login_server, EMPTY_GUID, _get_aad_token(cli_ctx,\n                                     ^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 285, in _get_aad_token\n    return _get_aad_token_after_challenge(cli_ctx,\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/azure/cli/command_modules/acr/_docker_utils.py\", line 159, in _get_aad_token_after_challenge\n    response = requests.post(url=authhost, data=urlencode(content), headers=headers,\n               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 115, in post\n    return request(\"post\", url, data=data, json=json, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/api.py\", line 59, in request\n    return session.request(method=method, url=url, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 589, in request\n    resp = self.send(prep, **send_kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/sessions.py\", line 703, in send\n    r = adapter.send(request, **kwargs)\n        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/lib64/az/lib/python3.12/site-packages/requests/adapters.py\", line 700, in send\n    raise ConnectionError(e, request=request)\nrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='arohcpsvcdev.azurecr.io', port=443): Max retries exceeded with url: /oauth2/exchange (Caused by NameResolutionError(\"HTTPSConnection(host='arohcpsvcdev.azurecr.io', port=443): Failed to resolve 'arohcpsvcdev.azurecr.io' ([Errno -2] Name or service not known)\"))\nTo check existing issues, please visit: https://github.com/Azure/azure-cli/issues\n exit status 1"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 38f15ef · merge 80403bb
 n/a provision failed ⚠#4407 (merged) 1
provision (1)
error running Shell Step, failed to build env vars: failed to get input values: variable SUBSCRIPTION_ID invalid: failed to lookup config reference mgmt.subscription.id for SUBSCRIPTION_ID: configuration[mgmt][subscription]: key id not found
10:21:47 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/delete-non-swift-user-nodepools · suite step graph
Show raw failure
time=2026-03-12T11:04:15.591Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=delete-non-swift-user-nodepools description="Step delete-non-swift-user-nodepools\n  Kind: Shell\n  Command: ./scripts/delete-user-nodepools.sh\n"
time=2026-03-12T11:04:17.912Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=delete-non-swift-user-nodepools err="error running Shell Step, failed to build env vars: failed to get input values: variable SUBSCRIPTION_ID invalid: failed to lookup config reference mgmt.subscription.id for SUBSCRIPTION_ID: configuration[mgmt][subscription]: key id not found"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 4a2f64c · merge ee63326
 n/a unknown failed #4428 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head afc3425 · merge e9ca15d
 n/a e2e failed #3907 (merged) 19
e2e (19)
Test Panicked
09:28:39 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000e1f4c0?, {0x7f69ecbdc088?, 0xc0008dc510?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc0009feea0, {0x7f69ecbdc088, 0xc0008dc510}, {0x7845f8a, 0x14}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func2.2({0x7f69ecbdc088, 0xc0008dc510})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/admin_api.go:60 +0xf1
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0008dc510?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 87
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000e1f4c0?, {0x7f69ecbdc088?, 0xc0008dc510?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc0009feea0, {0x7f69ecbdc088, 0xc0008dc510}, {0x7845f8a, 0x14}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func2.2({0x7f69ecbdc088, 0xc0008dc510})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/admin_api.go:60 +0xf1
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0008dc510?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 87
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000e2eb40?, {0x7f075ef70500?, 0xc000f8e0f0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000f88480, {0x7f075ef70500, 0xc000f8e0f0}, {0x78283bd, 0xe}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func23.2({0x7f075ef70500, 0xc000f8e0f0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:58 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000f8e0f0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 92
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000e2eb40?, {0x7f075ef70500?, 0xc000f8e0f0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000f88480, {0x7f075ef70500, 0xc000f8e0f0}, {0x78283bd, 0xe}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func23.2({0x7f075ef70500, 0xc000f8e0f0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:58 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000f8e0f0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 92
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc0007abec0?, {0x7fc25ed82060?, 0xc000a1f3b0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000a3a000, {0x7fc25ed82060, 0xc000a1f3b0}, {0x784adce, 0x15}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func18.2({0x7fc25ed82060, 0xc000a1f3b0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:67 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000a1f3b0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 46
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc0007abec0?, {0x7fc25ed82060?, 0xc000a1f3b0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000a3a000, {0x7fc25ed82060, 0xc000a1f3b0}, {0x784adce, 0x15}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func18.2({0x7fc25ed82060, 0xc000a1f3b0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:67 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000a1f3b0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 46
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000ea0540?, {0x7fd4ca675260?, 0xc0002679e0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000cc98c0, {0x7fd4ca675260, 0xc0002679e0}, {0x781f9d0, 0xc}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func8.2({0x7fd4ca675260, 0xc0002679e0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:54 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0002679e0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 92
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000ea0540?, {0x7fd4ca675260?, 0xc0002679e0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000cc98c0, {0x7fd4ca675260, 0xc0002679e0}, {0x781f9d0, 0xc}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func8.2({0x7fd4ca675260, 0xc0002679e0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:54 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0002679e0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 92
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc0007f1f00?, {0x7fc1704e0660?, 0xc000e06e40?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000e3c120, {0x7fc1704e0660, 0xc000e06e40}, {0x782cbf4, 0xf}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func23.3({0x7fc1704e0660, 0xc000e06e40})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:140 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000e06e40?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 95
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc0007f1f00?, {0x7fc1704e0660?, 0xc000e06e40?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000e3c120, {0x7fc1704e0660, 0xc000e06e40}, {0x782cbf4, 0xf}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func23.3({0x7fc1704e0660, 0xc000e06e40})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:140 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000e06e40?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 95
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000b4ed00?, {0x7ff728538688?, 0xc000461080?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc0003870e0, {0x7ff728538688, 0xc000461080}, {0x784ad3b, 0x15}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func5.1.1({0x7ff728538688, 0xc000461080})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:68 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000461080?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 96
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000b4ed00?, {0x7ff728538688?, 0xc000461080?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc0003870e0, {0x7ff728538688, 0xc000461080}, {0x784ad3b, 0x15}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func5.1.1({0x7ff728538688, 0xc000461080})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:68 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000461080?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 96
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000387580?, {0x7f6081ba0970?, 0xc0005f86f0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000ea5560, {0x7f6081ba0970, 0xc0005f86f0}, {0x78242d6, 0xd}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func16.2({0x7f6081ba0970, 0xc0005f86f0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:59 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0005f86f0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 44
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000387580?, {0x7f6081ba0970?, 0xc0005f86f0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000ea5560, {0x7f6081ba0970, 0xc0005f86f0}, {0x78242d6, 0xd}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func16.2({0x7f6081ba0970, 0xc0005f86f0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:59 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0005f86f0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 44
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000670c00?, {0x7f06383ece60?, 0xc000b81890?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc00082b8c0, {0x7f06383ece60, 0xc000b81890}, {0x7840c2f, 0x13}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func6.2({0x7f06383ece60, 0xc000b81890})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:64 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000b81890?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 87
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000670c00?, {0x7f06383ece60?, 0xc000b81890?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc00082b8c0, {0x7f06383ece60, 0xc000b81890}, {0x7840c2f, 0x13}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func6.2({0x7f06383ece60, 0xc000b81890})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:64 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000b81890?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 87
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000a74e80?, {0x7f3a4c080a60?, 0xc00095be00?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000d8c480, {0x7f3a4c080a60, 0xc00095be00}, {0x783172a, 0x10}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func4.2({0x7f3a4c080a60, 0xc00095be00})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:59 +0x105
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc00095be00?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 76
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000a74e80?, {0x7f3a4c080a60?, 0xc00095be00?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000d8c480, {0x7f3a4c080a60, 0xc00095be00}, {0x783172a, 0x10}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func4.2({0x7f3a4c080a60, 0xc00095be00})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:59 +0x105
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc00095be00?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 76
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc0007a4eb0?, {0x7f4fb44f2860?, 0xc000683080?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc00099d680, {0x7f4fb44f2860, 0xc000683080}, {0x7855240, 0x17}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func21.2({0x7f4fb44f2860, 0xc000683080})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:57 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000683080?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 65
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc0007a4eb0?, {0x7f4fb44f2860?, 0xc000683080?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc00099d680, {0x7f4fb44f2860, 0xc000683080}, {0x7855240, 0x17}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func21.2({0x7f4fb44f2860, 0xc000683080})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:57 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000683080?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 65
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000e10680?, {0x7fcd476d5f60?, 0xc000760330?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc0006f6d80, {0x7fcd476d5f60, 0xc000760330}, {0x78077be, 0x7}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func19.2({0x7fcd476d5f60, 0xc000760330})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:58 +0x155
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000760330?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 73
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000e10680?, {0x7fcd476d5f60?, 0xc000760330?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc0006f6d80, {0x7fcd476d5f60, 0xc000760330}, {0x78077be, 0x7}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func19.2({0x7fcd476d5f60, 0xc000760330})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:58 +0x155
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000760330?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 73
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000d76800?, {0x7fb1b0483c20?, 0xc000980570?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc00054d440, {0x7fb1b0483c20, 0xc000980570}, {0x7818172, 0xa}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func13.2.1({0x7fb1b0483c20, 0xc000980570})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:133 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000980570?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 101
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000d76800?, {0x7fb1b0483c20?, 0xc000980570?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc00054d440, {0x7fb1b0483c20, 0xc000980570}, {0x7818172, 0xa}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func13.2.1({0x7fb1b0483c20, 0xc000980570})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:133 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000980570?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 101
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000c47580?, {0x7fd2196f8908?, 0xc000228a20?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000a737a0, {0x7fd2196f8908, 0xc000228a20}, {0x78367f1, 0x11}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func25.1({0x7fd2196f8908, 0xc000228a20})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:58 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000228a20?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 100
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000c47580?, {0x7fd2196f8908?, 0xc000228a20?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000a737a0, {0x7fd2196f8908, 0xc000228a20}, {0x78367f1, 0x11}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func25.1({0x7fd2196f8908, 0xc000228a20})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:58 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000228a20?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 100
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to list available HCP OpenShift versions and validate response content · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).Get20240610ClientFactoryOrDie(0x78779c6?, {0x7efdc1169860?, 0xc000abd2c0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:617 +0x37
github.com/Azure/ARO-HCP/test/e2e.init.func1.1({0x7efdc1169860, 0xc000abd2c0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:38 +0x6f
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000abd2c0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 30
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).Get20240610ClientFactoryOrDie(0x78779c6?, {0x7efdc1169860?, 0xc000abd2c0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:617 +0x37
github.com/Azure/ARO-HCP/test/e2e.init.func1.1({0x7efdc1169860, 0xc000abd2c0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:38 +0x6f
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000abd2c0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 30
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000930c00?, {0x7f32f816e460?, 0xc000132810?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000b5f0e0, {0x7f32f816e460, 0xc000132810}, {0x782cbc7, 0xf}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func11.2({0x7f32f816e460, 0xc000132810})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:107 +0x20f
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000132810?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 101
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000930c00?, {0x7f32f816e460?, 0xc000132810?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000b5f0e0, {0x7f32f816e460, 0xc000132810}, {0x782cbc7, 0xf}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func11.2({0x7f32f816e460, 0xc000132810})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:107 +0x20f
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000132810?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 101
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc0007a9f40?, {0x7f77ddf96a20?, 0xc0006bb530?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000b310e0, {0x7f77ddf96a20, 0xc0006bb530}, {0x781819a, 0xa}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func22.2({0x7f77ddf96a20, 0xc0006bb530})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:56 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0006bb530?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 86
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc0007a9f40?, {0x7f77ddf96a20?, 0xc0006bb530?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000b310e0, {0x7f77ddf96a20, 0xc0006bb530}, {0x781819a, 0xa}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func22.2({0x7f77ddf96a20, 0xc0006bb530})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:56 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0006bb530?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 86
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000a69630?, {0x7ff55c23be20?, 0xc0006d8210?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000bf07e0, {0x7ff55c23be20, 0xc0006d8210}, {0x786881c, 0x1b}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func10.1({0x7ff55c23be20, 0xc0006d8210})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:49 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0006d8210?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 100
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000a69630?, {0x7ff55c23be20?, 0xc0006d8210?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000bf07e0, {0x7ff55c23be20, 0xc0006d8210}, {0x786881c, 0x1b}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func10.1({0x7ff55c23be20, 0xc0006d8210})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:49 +0xe9
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0006d8210?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 100
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc0009e39f0?, {0x7f4d61f3c230?, 0xc0007a6210?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000f1aa20, {0x7f4d61f3c230, 0xc0007a6210}, {0x7855229, 0x17}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func14.1({0x7f4d61f3c230, 0xc0007a6210})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:70 +0xf1
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0007a6210?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 140
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc0009e39f0?, {0x7f4d61f3c230?, 0xc0007a6210?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000f1aa20, {0x7f4d61f3c230, 0xc0007a6210}, {0x7855229, 0x17}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func14.1({0x7f4d61f3c230, 0xc0007a6210})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:70 +0xf1
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc0007a6210?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 140
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
Test Panicked
09:28:39 UTC · failed at e2e · test Customer should be able to list HCP clusters without node pools at both subscription and resource group levels · suite rp-api-compat-all/parallel
Show raw failure
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000a15ec0?, {0x7fba2bfcef58?, 0xc000b65ce0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000cad9e0, {0x7fba2bfcef58, 0xc000b65ce0}, {0x782cbb8, 0xf}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func9.1({0x7fba2bfcef58, 0xc000b65ce0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:55 +0x21e
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000b65ce0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 109
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
github.com/Azure/ARO-HCP/test/util/framework.Must[...](...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).GetARMResourcesClientFactoryOrDie(0xc000a15ec0?, {0x7fba2bfcef58?, 0xc000b65ce0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:609 +0x37
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).NewResourceGroup(0xc000cad9e0, {0x7fba2bfcef58, 0xc000b65ce0}, {0x782cbb8, 0xf}, {0xc000014109, 0x7})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:367 +0x2a5
github.com/Azure/ARO-HCP/test/e2e.init.func9.1({0x7fba2bfcef58, 0xc000b65ce0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_list_no_infra.go:55 +0x21e
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x82567a0?, 0xc000b65ce0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 109
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked: failed to get subscription ID: AzureCLICredential: signal: killed
fail [github.com/Azure/ARO-HCP/test/util/framework/per_invocation_framework.go:402]: Test Panicked
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a36f37a · merge a36f37a
 n/a n/a passed #4205 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head aa9e109
 n/a e2e failed ⚠#4281 (open) 2
e2e (2)
x509: malformed certificate
08:46:05 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:221]: cluster CA data must contain valid X.509 certificates
Unexpected error:
    <*errors.errorString | 0xc0010498e0>: 
    x509: malformed certificate
    {
        s: "x509: malformed certificate",
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:221]: cluster CA data must contain valid X.509 certificates
Unexpected error:
    <*errors.errorString | 0xc0010498e0>: 
    x509: malformed certificate
    {
        s: "x509: malformed certificate",
    }
occurred
whoami failed: Post "<url>
08:46:05 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:276]: Timed out after 30.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0011b0100>: 
    whoami failed: Post "https://sessiongate.j8509184.hcpsvc.osadev.cloud/sessiongate/breakglass-mrtw5/kas/apis/authentication.k8s.io/v1/selfsubjectreviews": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        msg: "whoami failed: Post \"https://sessiongate.j8509184.hcpsvc.osadev.cloud/sessiongate/breakglass-mrtw5/kas/apis/authentication.k8s.io/v1/selfsubjectreviews\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
        err: <*url.Error | 0xc00119b260>{
            Op: "Post",
            URL: "https://sessiongate.j8509184.hcpsvc.osadev.cloud/sessiongate/breakglass-mrtw5/kas/apis/authentication.k8s.io/v1/selfsubjectreviews",
            Err: <*tls.CertificateVerificationError | 0xc00119b230>{
                UnverifiedCertificates: [
                    {
                        Raw: [48, 130, 3, 163, 48, 130, 2, 139, 160, 3, 2, 1, 2, 2, 16, 3, 64, 236, 32, 244, 241, 76, 66, 157, 14, 246, 121, 236, 162, 69, 218, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 51, 49, 49, 48, 47, 6, 3, 85, 4, 3, 19, 40, 115, 101, 115, 115, 105, 111, 110, 103, 97, 116, 101, 46, 106, 56, 53, 48, 57, 49, 56, 52, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 48, 51, 49, 50, 48, 57, 49, 52, 53, 52, 90, 23, 13, 50, 55, 48, 51, 49, 50, 48, 57, 50, 52, 53, 52, 90, 48, 51, 49, 49, 48, 47, 6, 3, 85, 4, 3, 19, 40, 115, 101, 115, 115, 105, 111, 110, 103, 97, 116, 101, 46, 106, 56, 53, 48, 57, 49, 56, 52, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 189, 90, 48, 200, 73, 37, 164, 121, 122, 138, 179, 209, 50, 212, 239, 199, 61, 166, 221, 171, 94, 218, 183, 66, 66, 85, 64, 192, 252, 138, 101, 113, 116, 232, 198, 2, 92, 65, 41, 40, 20, 42, 20, 18, 180, 57, 200, 148, 102, 13, 229, 65, 161, 60, 169, 252, 89, 179, 156, 223, 201, 4, 10, 123, 239, 3, 231, 197, 0, 216, 231, 187, 156, 188, 130, 65, 186, 239, 94, 104, 231, 43, 181, 189, 179, 184, 114, 79, 184, 100, 152, 55, 196, 185, 150, 41, 0, 144, 173, 131, 162, 139, 159, 52, 170, 103, 29, 9, 161, 83, 93, 190, 206, 125, 182, 135, 199, 121, 138, 71, 144, 40, 142, 136, 255, 45, 223, 64, 158, 50, 9, 219, 39, 1, 124, 179, 138, 55, 116, 44, 104, 52, 46, 178, 13, 13, 200, 114, 59, 209, 251, 189, 254, 198, 123, 191, 46, 239, 33, 3, 150, 111, 122, 15, 177, 214, 181, 158, 145, 10, 139, 248, 54, 102, 189, 15, 38, 198, 55, 26, 7, 181, 25, 13, 239, 154, 82, 37, 64, 43, 244, 7, 198, 4, 240, 131, 161, 214, 200, 225, 34, 72, 246, 216, 120, 31, 90, 78, 199, 228, 254, 161, 212, 43, 22, 55, 174, 10, 12, 240, 96, 59, 89, 184, 187, 45, 158, 57, 142, 200, 214, 6, 97, 123, 111, 209, 157, 30, 171, 214, 23, 93, 226, 26, 206, 253, 164, 23, 51, 188, 7, 184, 187, 60, 210, 25, 2, 3, 1, 0, 1, 163, 129, 178, 48, 129, 175, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 51, 6, 3, 85, 29, 17, 4, 44, 48, 42, 130, 40, 115, 101, 115, 115, 105, 111, 110, 103, 97, 116, 101, 46, 106, 56, 53, 48, 57, 49, 56, 52, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 176, 118, 174, 101, 221, 85, 31, 232, 79, 124, 151, 22, 150, 147, 35, 77, 80, 170, 85, 237, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 176, 118, 174, 101, 221, 85, 31, 232, 79, 124, 151, 22, 150, 147, 35, 77, 80, 170, 85, 237, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 22, 195, 82, 20, 227, 93, 180, 69, 100, 13, 157, 56, 89, 197, 176, 49, 218, 196, 4, 38, 54, 46, 126, 136, 113, 92, 86, 10, 177, 39, 81, 219, 240, 173, 87, 174, 14, 26, 130, 227, 193, 115, 155, 163, 189, 221, 46, 34, 160, 111, 250, 238, 248, 11, 212, 22, 74, 28, 7, 168, 32, 110, 194, 233, 209, 235, 254, 117, 138, 198, 45, 155, 227, 174, 155, 24, 95, 166, 84, 250, 35, 58, 5, 113, 116, 16, 8, 160, 212, 138, 177, 121, 74, 99, 42, 227, 26, 240, 103, 114, 208, 165, 63, 253, 1...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:276]: Timed out after 30.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0011b0100>: 
    whoami failed: Post "https://sessiongate.j8509184.hcpsvc.osadev.cloud/sessiongate/breakglass-mrtw5/kas/apis/authentication.k8s.io/v1/selfsubjectreviews": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        msg: "whoami failed: Post \"https://sessiongate.j8509184.hcpsvc.osadev.cloud/sessiongate/breakglass-mrtw5/kas/apis/authentication.k8s.io/v1/selfsubjectreviews\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
        err: <*url.Error | 0xc00119b260>{
            Op: "Post",
            URL: "https://sessiongate.j8509184.hcpsvc.osadev.cloud/sessiongate/breakglass-mrtw5/kas/apis/authentication.k8s.io/v1/selfsubjectreviews",
            Err: <*tls.CertificateVerificationError | 0xc00119b230>{
                UnverifiedCertificates: [
                    {
                        Raw: [48, 130, 3, 163, 48, 130, 2, 139, 160, 3, 2, 1, 2, 2, 16, 3, 64, 236, 32, 244, 241, 76, 66, 157, 14, 246, 121, 236, 162, 69, 218, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 51, 49, 49, 48, 47, 6, 3, 85, 4, 3, 19, 40, 115, 101, 115, 115, 105, 111, 110, 103, 97, 116, 101, 46, 106, 56, 53, 48, 57, 49, 56, 52, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 30, 23, 13, 50, 54, 48, 51, 49, 50, 48, 57, 49, 52, 53, 52, 90, 23, 13, 50, 55, 48, 51, 49, 50, 48, 57, 50, 52, 53, 52, 90, 48, 51, 49, 49, 48, 47, 6, 3, 85, 4, 3, 19, 40, 115, 101, 115, 115, 105, 111, 110, 103, 97, 116, 101, 46, 106, 56, 53, 48, 57, 49, 56, 52, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 189, 90, 48, 200, 73, 37, 164, 121, 122, 138, 179, 209, 50, 212, 239, 199, 61, 166, 221, 171, 94, 218, 183, 66, 66, 85, 64, 192, 252, 138, 101, 113, 116, 232, 198, 2, 92, 65, 41, 40, 20, 42, 20, 18, 180, 57, 200, 148, 102, 13, 229, 65, 161, 60, 169, 252, 89, 179, 156, 223, 201, 4, 10, 123, 239, 3, 231, 197, 0, 216, 231, 187, 156, 188, 130, 65, 186, 239, 94, 104, 231, 43, 181, 189, 179, 184, 114, 79, 184, 100, 152, 55, 196, 185, 150, 41, 0, 144, 173, 131, 162, 139, 159, 52, 170, 103, 29, 9, 161, 83, 93, 190, 206, 125, 182, 135, 199, 121, 138, 71, 144, 40, 142, 136, 255, 45, 223, 64, 158, 50, 9, 219, 39, 1, 124, 179, 138, 55, 116, 44, 104, 52, 46, 178, 13, 13, 200, 114, 59, 209, 251, 189, 254, 198, 123, 191, 46, 239, 33, 3, 150, 111, 122, 15, 177, 214, 181, 158, 145, 10, 139, 248, 54, 102, 189, 15, 38, 198, 55, 26, 7, 181, 25, 13, 239, 154, 82, 37, 64, 43, 244, 7, 198, 4, 240, 131, 161, 214, 200, 225, 34, 72, 246, 216, 120, 31, 90, 78, 199, 228, 254, 161, 212, 43, 22, 55, 174, 10, 12, 240, 96, 59, 89, 184, 187, 45, 158, 57, 142, 200, 214, 6, 97, 123, 111, 209, 157, 30, 171, 214, 23, 93, 226, 26, 206, 253, 164, 23, 51, 188, 7, 184, 187, 60, 210, 25, 2, 3, 1, 0, 1, 163, 129, 178, 48, 129, 175, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 9, 6, 3, 85, 29, 19, 4, 2, 48, 0, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 48, 51, 6, 3, 85, 29, 17, 4, 44, 48, 42, 130, 40, 115, 101, 115, 115, 105, 111, 110, 103, 97, 116, 101, 46, 106, 56, 53, 48, 57, 49, 56, 52, 46, 104, 99, 112, 115, 118, 99, 46, 111, 115, 97, 100, 101, 118, 46, 99, 108, 111, 117, 100, 48, 31, 6, 3, 85, 29, 35, 4, 24, 48, 22, 128, 20, 176, 118, 174, 101, 221, 85, 31, 232, 79, 124, 151, 22, 150, 147, 35, 77, 80, 170, 85, 237, 48, 29, 6, 3, 85, 29, 14, 4, 22, 4, 20, 176, 118, 174, 101, 221, 85, 31, 232, 79, 124, 151, 22, 150, 147, 35, 77, 80, 170, 85, 237, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 22, 195, 82, 20, 227, 93, 180, 69, 100, 13, 157, 56, 89, 197, 176, 49, 218, 196, 4, 38, 54, 46, 126, 136, 113, 92, 86, 10, 177, 39, 81, 219, 240, 173, 87, 174, 14, 26, 130, 227, 193, 115, 155, 163, 189, 221, 46, 34, 160, 111, 250, 238, 248, 11, 212, 22, 74, 28, 7, 168, 32, 110, 194, 233, 209, 235, 254, 117, 138, 198, 45, 155, 227, 174, 155, 24, 95, 166, 84, 250, 35, 58, 5, 113, 116, 16, 8, 160, 212, 138, 177, 121, 74, 99, 42, 227, 26, 240, 103, 114, 208, 165, 63, 253, 1...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c117b0a · merge c117b0a
 n/a n/a passed #4350 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 0b33790 · merge 0b33790
 n/a n/a passed #4376 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0cc8e93
 n/a provision failed ⚠#4381 (closed) 1
provision (1)
error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform "Exists" on destination: HEAD "<url>": GET "<url>": response status code 401: unauthorized: authentication required, visit <url> for more
07:46:16 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/global/mirror-image · suite step graph
Show raw failure
time=2026-03-12T08:33:18.623Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T08:33:19.119Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-12T08:33:19.774Z level=INFO msg="No credentials found in pull secret for source registry, using anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image registry=registry.k8s.io
time=2026-03-12T08:33:19.774Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-12T08:33:22.061Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image source=registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 target=arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5
time=2026-03-12T08:33:22.293Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: e68d27ab-87e3-454e-b50a-979e6a65df61"
time=2026-03-12T08:33:52.389Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 3caba028-50d6-4629-bda5-82b8cef2c9e5"
time=2026-03-12T08:34:22.493Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 7fab28c5-bb45-4b2f-93b9-ad685d614fdf"
time=2026-03-12T08:35:22.493Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T08:35:23.085Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-12T08:35:23.421Z level=INFO msg="No credentials found in pull secret for source registry, using anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image registry=registry.k8s.io
time=2026-03-12T08:35:23.421Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-12T08:35:25.849Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image source=registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 target=arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5
time=2026-03-12T08:35:25.950Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 58a965ac-5c93-4ac6-94c5-0a299e44958c"
time=2026-03-12T08:35:56.065Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: e2056d6b-ef08-441f-ad35-46819df94410"
time=2026-03-12T08:36:26.175Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 91bf5abe-cfa6-44d3-9085-a7b6000705af"
time=2026-03-12T08:37:26.179Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T08:37:26.598Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-12T08:37:26.955Z level=INFO msg="No credentials found in pull secret for source registry, using anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image registry=registry.k8s.io
time=2026-03-12T08:37:26.955Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-12T08:37:28.583Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image source=registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 target=arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5
time=2026-03-12T08:37:28.682Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: db748e17-bb53-4d9b-ad0f-52aad91ee58d"
time=2026-03-12T08:37:58.786Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: b4bec68c-bf7d-4f77-a7d9-fe02e51d0cda"
time=2026-03-12T08:38:28.893Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 1dc234d4-4319-41e0-853e-72a10a0c9beb"
time=2026-03-12T08:39:28.893Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T08:39:29.248Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-12T08:39:29.612Z level=INFO msg="No credentials found in pull secret for source registry, using anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image registry=registry.k8s.io
time=2026-03-12T08:39:29.612Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-12T08:39:31.160Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image source=registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 target=arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5
time=2026-03-12T08:39:31.261Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 8520dfb4-1e81-4700-a74f-9c680527dc8e"
time=2026-03-12T08:40:01.449Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: ca1cb04d-8a0e-4932-a6d0-c4da0c5c00b9"
time=2026-03-12T08:40:31.551Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: ca27ba07-125f-4afd-a62b-30f32138d107"
time=2026-03-12T08:41:31.552Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T08:41:32.115Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-12T08:41:32.456Z level=INFO msg="No credentials found in pull secret for source registry, using anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image registry=registry.k8s.io
time=2026-03-12T08:41:32.456Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-12T08:41:34.440Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image source=registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 target=arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5
time=2026-03-12T08:41:34.556Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 2869028f-a756-4655-afd3-40587deece02"
time=2026-03-12T08:42:04.664Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 7be1b40b-0ac1-4b29-98c0-ea1d465ca44b"
time=2026-03-12T08:42:34.801Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 5f4aaa00-7089-4375-9851-388274a6e7d5"
time=2026-03-12T08:43:34.801Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 5f4aaa00-7089-4375-9851-388274a6e7d5"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 79860ed · merge 61fd736
 n/a e2e failed ★#4409 (merged) 1
e2e (1)
Yet, Eventually failed because the matcher was not satisfied:
06:53:50 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:318]: Told to stop trying after 20.249s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
    <v1.PodPhase>: Pending
to equal
    <v1.PodPhase>: Succeeded
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:318]: Told to stop trying after 20.249s.
Pod failed; see logs for details
At one point, however, the function did return successfully.
Yet, Eventually failed because the matcher was not satisfied:
Expected
    <v1.PodPhase>: Pending
to equal
    <v1.PodPhase>: Succeeded
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head f081373 · merge f081373
 n/a n/a passed #4427 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head f91e23c · merge f91e23c
 n/a n/a passed #4410 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 2cb1dbe
 n/a provision failed ⚠#4381 (closed) 1
provision (1)
error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform "Exists" on destination: HEAD "<url>": GET "<url>": response status code 401: unauthorized: authentication required, visit <url> for more
01:46:24 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/global/mirror-image · suite step graph
Show raw failure
time=2026-03-12T02:37:43.436Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T02:37:43.846Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-12T02:37:44.409Z level=INFO msg="No credentials found in pull secret for source registry, using anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image registry=registry.k8s.io
time=2026-03-12T02:37:44.409Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-12T02:37:46.324Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image source=registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 target=arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5
time=2026-03-12T02:37:46.567Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: ae068a40-2ba6-4362-9812-2edfe309bbee"
time=2026-03-12T02:38:16.870Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: b2df11e8-61fa-4fb5-9b72-f0e6bd79f604"
time=2026-03-12T02:38:46.984Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 895103c4-016b-403a-be27-a9bfa8c030c8"
time=2026-03-12T02:39:46.985Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T02:39:47.365Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-12T02:39:47.716Z level=INFO msg="No credentials found in pull secret for source registry, using anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image registry=registry.k8s.io
time=2026-03-12T02:39:47.716Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-12T02:39:49.440Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image source=registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 target=arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5
time=2026-03-12T02:39:49.531Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 7d50c92f-68fc-4bb2-be81-3377044f424c"
time=2026-03-12T02:40:19.622Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: a290f7e9-b177-47fc-8180-2e0680c0ebee"
time=2026-03-12T02:40:49.717Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 8f2af50e-827c-4931-94cb-4f02a4a7f374"
time=2026-03-12T02:41:49.718Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T02:41:50.103Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-12T02:41:50.450Z level=INFO msg="No credentials found in pull secret for source registry, using anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image registry=registry.k8s.io
time=2026-03-12T02:41:50.451Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-12T02:41:52.083Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image source=registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 target=arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5
time=2026-03-12T02:41:52.177Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 0f4f10c8-31db-4b88-b1c4-359c61d7ebaa"
time=2026-03-12T02:42:22.332Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 850e9966-c3e2-42ef-82e0-4b83d529ff2d"
time=2026-03-12T02:42:52.427Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: edce3a43-3792-4edd-8c93-d35a75cae0bd"
time=2026-03-12T02:43:52.428Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T02:43:52.836Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-12T02:43:53.178Z level=INFO msg="No credentials found in pull secret for source registry, using anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image registry=registry.k8s.io
time=2026-03-12T02:43:53.178Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-12T02:43:54.931Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image source=registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 target=arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5
time=2026-03-12T02:43:55.029Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 2c30f8ed-db23-4fb0-92c5-32ba5ae29642"
time=2026-03-12T02:44:25.134Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 5d4b9f84-1387-4474-bd0c-4f1445d6a31f"
time=2026-03-12T02:44:55.252Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 4d37a609-5317-4b55-9135-f25fe88a4174"
time=2026-03-12T02:45:55.253Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> secretSyncController.image.registry <nil>}:{<nil> secretSyncController.image.repository <nil>}@{<nil> secretSyncController.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-03-12T02:45:55.698Z level=INFO msg="Fetching pull secret from Key Vault" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image vault=arohcpdev-global secret=component-sync-pull-secret
time=2026-03-12T02:45:56.035Z level=INFO msg="No credentials found in pull secret for source registry, using anonymous access" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image registry=registry.k8s.io
time=2026-03-12T02:45:56.035Z level=INFO msg="Logging into target ACR" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image acr=arohcpsvcdev
time=2026-03-12T02:45:57.949Z level=INFO msg="Mirroring image" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image source=registry.k8s.io/secrets-store-sync/controller@sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5 target=arohcpsvcdev.azurecr.io/secrets-store-sync/controller:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5
time=2026-03-12T02:45:58.043Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=1 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 0418e518-c99d-4a88-a5cb-670d772ae436"
time=2026-03-12T02:46:28.145Z level=INFO msg="Image copy failed, retrying" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image attempt=2 error="failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: 892e9b18-67bb-438e-90c5-3e7479c567f1"
time=2026-03-12T02:46:58.237Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image step=mirror-image error="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: b500b1c4-3f1a-408a-95fc-278c9978e37f"
time=2026-03-12T02:47:58.237Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to mirror image after 3 attempts: failed to copy image: failed to perform \"Exists\" on destination: HEAD \"https://arohcpsvcdev.azurecr.io/v2/secrets-store-sync/controller/manifests/sha256:15768ff940ba7744d9994f8abde9ceb3ec3c5c6a024cb1256feb4d1b17c6b9f5\": GET \"https://arohcpsvcdev.azurecr.io/oauth2/token?scope=repository%3Asecrets-store-sync%2Fcontroller%3Apull&service=arohcpsvcdev.azurecr.io\": response status code 401: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information. CorrelationId: b500b1c4-3f1a-408a-95fc-278c9978e37f"
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head e5aa47f · merge 8550ece
 n/a unknown failed #4219 (merged) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head e0b528c · merge e0b528c
 n/a n/a passed #4371 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
batch
head 1892906 · merge 1892906
 n/a n/a passed #3820 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head f91e23c · merge f91e23c
 n/a n/a passed #4410 (merged) 0 n/a
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head fb374b2
 n/a unknown failed #4424 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 89e2e4b
 n/a unknown failed #4423 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head dfea60a
 n/a unknown failed #4422 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 1594ac0
 n/a unknown failed #4421 (closed) 1 non-artifact-backed failure (no junit artifacts)
n/a pull-ci-Azure-ARO-HCP-main-e2e-parallel
head ad4a391
 n/a unknown failed #4420 (closed) 1 non-artifact-backed failure (no junit artifacts)