CIHealth
<
Weekly: Mar 23 - Mar 29▾
Last 7 Days Weekly: Mar 23 - Mar 29 Sprint: Mar 16 - Mar 29 Single Day: Mar 29
Custom time window
> Reset
View JSON API

Environment: INT

Failure patterns: 13 · Failures matched: 326 · Total runs: 42 · Runs affected: 22

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
e2e1228.57%12 of 42 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsMar 22: 1 · Mar 23: 0 · Mar 24: 19 · Mar 25: 33 · Mar 26: 80 · Mar 27: 57 · Mar 28: 0 · Mar 29: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014cacc0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-ck2bxp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-ck2bxp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000929890>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014cacc0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-ck2bxp" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-ck2bxp\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000929890>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-ck2bxp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012dc8d0>: 
    failed to create HCP cluster ea-listk99r75, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: failed waiting for cluster="ea-listk99r75" in resourcegroup="ea-list-7tsrmq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-listk99r75, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: failed waiting for cluster=\"ea-listk99r75\" in resourcegroup=\"ea-list-7tsrmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001212330>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq",
            },
            <*fmt.wrapErrors | 0xc0012dc8a0>{
                msg: "failed waiting for cluster=\"ea-listk99r75\" in resourcegroup=\"ea-list-7tsrmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001212330>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc0012dc8d0>: 
    failed to create HCP cluster ea-listk99r75, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: failed waiting for cluster="ea-listk99r75" in resourcegroup="ea-list-7tsrmq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster ea-listk99r75, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: failed waiting for cluster=\"ea-listk99r75\" in resourcegroup=\"ea-list-7tsrmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001212330>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq",
            },
            <*fmt.wrapErrors | 0xc0012dc8a0>{
                msg: "failed waiting for cluster=\"ea-listk99r75\" in resourcegroup=\"ea-list-7tsrmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001212330>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listk99r75 in resource group ea-list-7tsrmq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009fc450>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jnxjrh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jnxjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000c71e70>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh",
            },
            <*fmt.wrapErrors | 0xc0009fc420>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jnxjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000c71e70>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
    <*fmt.wrapErrors | 0xc0009fc450>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jnxjrh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jnxjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000c71e70>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh",
            },
            <*fmt.wrapErrors | 0xc0009fc420>{
                msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-jnxjrh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000c71e70>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-jnxjrh",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Contributing tests (26)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources11
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.1910
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelHCP Nodepools GPU instances creates and deletes vm type NC4asT4v3 in a single cluster10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config9
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets9
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL9
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request9
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to update node pool labels and taints8
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelControl plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.212
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelControl plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should upgrade a nodepool from 4.20.z to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should upgrade a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (12)
Date (UTC)Associated PRProw job
n/a2037557559711240192
n/a2037452645840457728
n/a2037413674255126528
n/a2037366470257676288
n/a2037303065828134912
n/a2037210183100272640
n/a2037065115634765824
n/a2037024456240533504
n/a2036978293944619008
n/a2036937715248795648
n/a2036888953293901824
n/a2036428866838335488
Interrupted by User
full failure pattern:
Interrupted by User
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
e2e1126.19%11 of 42 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 4 · Mar 25: 12 · Mar 26: 23 · Mar 27: 17 · Mar 28: 0 · Mar 29: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:68]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:68]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:127]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:127]: Interrupted by User
Contributing tests (13)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelControl plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.2010
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group10
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelControl plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.219
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling8
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled6
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelControl plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.191
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (11)
Date (UTC)Associated PRProw job
n/a2037452645840457728
n/a2037413674255126528
n/a2037366470257676288
n/a2037303065828134912
n/a2037210183100272640
n/a2037065115634765824
n/a2037024456240533504
n/a2036978293944619008
n/a2036937715248795648
n/a2036888953293901824
n/a2036428866838335488
Cluster should become ready within 45 minutes
full failure pattern:
Cluster should become ready within 45 minutes
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e1023.81%10 of 42 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 1 · Mar 25: 2 · Mar 26: 4 · Mar 27: 3 · Mar 28: 0 · Mar 29: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.002s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.002s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.000s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.003s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.003s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle10
Affected runs (10)
Date (UTC)Associated PRProw job
n/a2037452645840457728
n/a2037413674255126528
n/a2037366470257676288
n/a2037210183100272640
n/a2037065115634765824
n/a2037024456240533504
n/a2036978293944619008
n/a2036937715248795648
n/a2036888953293901824
n/a2036428866838335488
Deserializaion Error: no output from command
full failure pattern:
Deserializaion Error: no output from command
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e716.67%7 of 42 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 2 · Mar 26: 3 · Mar 27: 2 · Mar 28: 0 · Mar 29: 1none
Full failure examples (3)
e/proc.go:466
runtime.semacquire1(0xc001686608, 0x0, 0x1, 0x0, 0x19)
	/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc000a595f8 sp=0xc000a59590 pc=0x2b3451d
sync.runtime_SemacquireWaitGroup(0xc000a30c20?, 0x40?)
	/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc000a59630 sp=0xc000a595f8 pc=0x2b59bae
sync.(*WaitGroup).Wait(0xc001686600)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc000a59658 sp=0xc000a59630 pc=0x2b7d0e5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc000548500, {0x7f9abcaaa958, 0xc00160ca50}, {{0xc000daf270, 0x1, 0x1}, 0x34630b8a000, {0x7995009, 0x8}})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:308 +0x273 fp=0xc000a59768 sp=0xc000a59658 pc=0x4fdc233
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000548500, {0x7f9abcaaa958, 0xc00160ca50})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:227 +0x225 fp=0xc000a59888 sp=0xc000a59768 pc=0x4fdbac5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f9abcaaa958?, 0xc00160ca50?})
	<autogenerated>:1 +0x33 fp=0xc000a598b0 sp=0xc000a59888 pc=0x4fea753
runtime.call16(0xc0011e87e0, 0xc000091030, 0x0, 0x0, 0x0, 0x10, 0xc000a59b08)
	/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc000a598d0 sp=0xc000a598b0 pc=0x2b5f163
runtime.reflectcall(0x73e36c0?, 0xc00160ca50?, 0x2?, 0x79c5cd8?, 0x0?, 0x12?, 0x7018be0?)
	<autogenerated>:1 +0x36 fp=0xc000a59910 sp=0xc000a598d0 pc=0x2b679f6
reflect.Value.call({0x699dce0?, 0xc000091030?, 0xc00000029f?}, {0x7989dfb, 0x4}, {0xc000e30f60, 0x1, 0x2ae7368?})
	/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc000a59e70 sp=0xc000a59910 pc=0x2b9a9c6
reflect.Value.Call({0x699dce0?, 0xc000091030?, 0x0?}, {0xc000e30f60?, 0x326f4c8?, 0xc000e00150?})
	/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc000a59ee8 sp=0xc000a59e70 pc=0x2b99ab9
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x8400760?, 0xc00160ca50?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc000a59f88 sp=0xc000a59ee8 pc=0x2fba290
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000a59fe0 sp=0xc000a59f88 pc=0x2fccd5b
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000a59fe8 sp=0xc000a59fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 104
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

goroutine 1180 gp=0xc00137efc0 m=nil [select]:
runtime.gopark(0xc000095ab8?, 0x2?, 0x80?, 0xfa?, 0xc000095aac?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000095938 sp=0xc000095918 pc=0x2b5848e
runtime.selectgo(0xc000095ab8, 0xc000095aa8, 0xc0016104e0?, 0x0, 0x33570dc?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000095a78 sp=0xc000095938 pc=0x2b33277
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x83ea848, 0xc000e9f4f0}, 0x79a98af?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000095ae8 sp=0xc000095a78 pc=0x3356836
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x842fc60, {0x83ea848, 0xc000e9f4f0}, 0xc00095ec50)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x732 fp=0xc000095de8 sp=0xc000095ae8 pc=0x3792ed2
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster({0x83ea848, 0xc000e9f4a0}, 0xc001997380, {0xc000d69100, 0x1a}, {0xc0017bae70, 0x18}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:201 +0x218 fp=0xc000095ef0 sp=0xc000095de8 pc=0x4fbdf38
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:349 +0x94 fp=0xc000095f78 sp=0xc000095ef0 pc=0x4fc0534
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc000095fe0 sp=0xc000095f78 pc=0x3874450
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1178
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xfffffffffffffffc
rbx    0x5
rcx    0x2ad7e2e
rdx    0x80
rdi    0x5
rsi    0x7ffe140e4e5c
rbp    0x7ffe140e4e10
rsp    0x7ffe140e4dc8
r8     0x0
r9     0x0
r10    0x432
r11    0x246
r12    0x7ffe140e4ea0
r13    0x1
r14    0xb57b700
r15    0x7
rip    0x2ad7e2e
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
e/proc.go:466
runtime.semacquire1(0xc001686608, 0x0, 0x1, 0x0, 0x19)
	/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc000a595f8 sp=0xc000a59590 pc=0x2b3451d
sync.runtime_SemacquireWaitGroup(0xc000a30c20?, 0x40?)
	/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc000a59630 sp=0xc000a595f8 pc=0x2b59bae
sync.(*WaitGroup).Wait(0xc001686600)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc000a59658 sp=0xc000a59630 pc=0x2b7d0e5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc000548500, {0x7f9abcaaa958, 0xc00160ca50}, {{0xc000daf270, 0x1, 0x1}, 0x34630b8a000, {0x7995009, 0x8}})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:308 +0x273 fp=0xc000a59768 sp=0xc000a59658 pc=0x4fdc233
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000548500, {0x7f9abcaaa958, 0xc00160ca50})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:227 +0x225 fp=0xc000a59888 sp=0xc000a59768 pc=0x4fdbac5
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f9abcaaa958?, 0xc00160ca50?})
	<autogenerated>:1 +0x33 fp=0xc000a598b0 sp=0xc000a59888 pc=0x4fea753
runtime.call16(0xc0011e87e0, 0xc000091030, 0x0, 0x0, 0x0, 0x10, 0xc000a59b08)
	/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc000a598d0 sp=0xc000a598b0 pc=0x2b5f163
runtime.reflectcall(0x73e36c0?, 0xc00160ca50?, 0x2?, 0x79c5cd8?, 0x0?, 0x12?, 0x7018be0?)
	<autogenerated>:1 +0x36 fp=0xc000a59910 sp=0xc000a598d0 pc=0x2b679f6
reflect.Value.call({0x699dce0?, 0xc000091030?, 0xc00000029f?}, {0x7989dfb, 0x4}, {0xc000e30f60, 0x1, 0x2ae7368?})
	/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc000a59e70 sp=0xc000a59910 pc=0x2b9a9c6
reflect.Value.Call({0x699dce0?, 0xc000091030?, 0x0?}, {0xc000e30f60?, 0x326f4c8?, 0xc000e00150?})
	/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc000a59ee8 sp=0xc000a59e70 pc=0x2b99ab9
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x8400760?, 0xc00160ca50?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc000a59f88 sp=0xc000a59ee8 pc=0x2fba290
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000a59fe0 sp=0xc000a59f88 pc=0x2fccd5b
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000a59fe8 sp=0xc000a59fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 104
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

goroutine 1180 gp=0xc00137efc0 m=nil [select]:
runtime.gopark(0xc000095ab8?, 0x2?, 0x80?, 0xfa?, 0xc000095aac?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000095938 sp=0xc000095918 pc=0x2b5848e
runtime.selectgo(0xc000095ab8, 0xc000095aa8, 0xc0016104e0?, 0x0, 0x33570dc?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000095a78 sp=0xc000095938 pc=0x2b33277
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x83ea848, 0xc000e9f4f0}, 0x79a98af?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000095ae8 sp=0xc000095a78 pc=0x3356836
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x842fc60, {0x83ea848, 0xc000e9f4f0}, 0xc00095ec50)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x732 fp=0xc000095de8 sp=0xc000095ae8 pc=0x3792ed2
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster({0x83ea848, 0xc000e9f4a0}, 0xc001997380, {0xc000d69100, 0x1a}, {0xc0017bae70, 0x18}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:201 +0x218 fp=0xc000095ef0 sp=0xc000095de8 pc=0x4fbdf38
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:349 +0x94 fp=0xc000095f78 sp=0xc000095ef0 pc=0x4fc0534
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc000095fe0 sp=0xc000095f78 pc=0x3874450
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1178
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xfffffffffffffffc
rbx    0x5
rcx    0x2ad7e2e
rdx    0x80
rdi    0x5
rsi    0x7ffe140e4e5c
rbp    0x7ffe140e4e10
rsp    0x7ffe140e4dc8
r8     0x0
r9     0x0
r10    0x432
r11    0x246
r12    0x7ffe140e4ea0
r13    0x1
r14    0xb57b700
r15    0x7
rip    0x2ad7e2e
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
ternal.extractBodyFunction.func2({0x83ff000?, 0xc000852900?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e fp=0xc000095f88 sp=0xc000095f58 pc=0x2fb863e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000095fe0 sp=0xc000095f88 pc=0x2fccd5b
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 110
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

goroutine 154 gp=0xc0002d6e00 m=nil [IO wait]:
runtime.gopark(0xc00128de20?, 0x202dc100?, 0x50?, 0x27?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000392720 sp=0xc000392700 pc=0x2b5848e
runtime.netpollblock(0x2bc2038?, 0x2ae4066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000392758 sp=0xc000392720 pc=0x2b17f37
internal/poll.runtime_pollWait(0x7f8021c36e00, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000392778 sp=0xc000392758 pc=0x2b56d25
internal/poll.(*pollDesc).wait(0xc000b26480?, 0xc000f12000?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0003927a0 sp=0xc000392778 pc=0x2be4d67
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000b26480, {0xc000f12000, 0x3500, 0x3500})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc000392838 sp=0xc0003927a0 pc=0x2be6199
net.(*netFD).Read(0xc000b26480, {0xc000f12000?, 0xc000f122c2?, 0x5?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc000392880 sp=0xc000392838 pc=0x2da9785
net.(*conn).Read(0xc0000bb868, {0xc000f12000?, 0x7f802021c948?, 0x7f806881f5c0?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc0003928c8 sp=0xc000392880 pc=0x2dbb825
crypto/tls.(*atLeastReader).Read(0xc000b105a0, {0xc000f12000?, 0x3239?, 0x2b551c5?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc000392910 sp=0xc0003928c8 pc=0x2e272bb
bytes.(*Buffer).ReadFrom(0xc000533ea8, {0x834e6c0, 0xc000b105a0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000392968 sp=0xc000392910 pc=0x2c22d18
crypto/tls.(*Conn).readFromUntil(0xc000533c08, {0x834f220, 0xc0000bb868}, 0xc0003929d0?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0003929a0 sp=0xc000392968 pc=0x2e2749e
crypto/tls.(*Conn).readRecordOrCCS(0xc000533c08, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000392c18 sp=0xc0003929a0 pc=0x2e2451b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000533c08, {0xc001591000, 0x1000, 0x3267100?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000392c80 sp=0xc000392c18 pc=0x2e2ae05
bufio.(*Reader).Read(0xc001455a40, {0xc0012bf004, 0x9, 0x32751ae?})
	/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc000392cb8 sp=0xc000392c80 pc=0x2cbe4d7
io.ReadAtLeast({0x834dce0, 0xc001455a40}, {0xc0012bf004, 0x9, 0x9}, 0x9)
	/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000392d00 sp=0xc000392cb8 pc=0x2bb2dce
io.ReadFull(...)
	/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0012bf004, 0x9, 0xc00000018b?}, {0x834dce0?, 0xc001455a40?})
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc000392d50 sp=0xc000392d00 pc=0x325dec5
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0012befc0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc000392dd8 sp=0xc000392d50 pc=0x325ea8b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0012befc0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc000392e18 sp=0xc000392dd8 pc=0x325f1f8
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000392fa8)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc000392f60 sp=0xc000392e18 pc=0x32720ea
golang.org/x/net/http2.(*ClientConn).readLoop(0xc001049dc0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc000392fc8 sp=0xc000392f60 pc=0x3271472
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc000392fe0 sp=0xc000392fc8 pc=0x326ac05
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000392fe8 sp=0xc000392fe0 pc=0x2b60c01
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 153
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5

rax    0xca
rbx    0x0
rcx    0x2b62a23
rdx    0x0
rdi    0xb57b7d0
rsi    0x80
rbp    0x7ffef1febfe0
rsp    0x7ffef1febf98
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xb5796e0
r15    0x3
rip    0x2b62a21
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
ternal.extractBodyFunction.func2({0x83ff000?, 0xc000852900?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e fp=0xc000095f88 sp=0xc000095f58 pc=0x2fb863e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000095fe0 sp=0xc000095f88 pc=0x2fccd5b
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 110
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

goroutine 154 gp=0xc0002d6e00 m=nil [IO wait]:
runtime.gopark(0xc00128de20?, 0x202dc100?, 0x50?, 0x27?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000392720 sp=0xc000392700 pc=0x2b5848e
runtime.netpollblock(0x2bc2038?, 0x2ae4066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc000392758 sp=0xc000392720 pc=0x2b17f37
internal/poll.runtime_pollWait(0x7f8021c36e00, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc000392778 sp=0xc000392758 pc=0x2b56d25
internal/poll.(*pollDesc).wait(0xc000b26480?, 0xc000f12000?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0003927a0 sp=0xc000392778 pc=0x2be4d67
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc000b26480, {0xc000f12000, 0x3500, 0x3500})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc000392838 sp=0xc0003927a0 pc=0x2be6199
net.(*netFD).Read(0xc000b26480, {0xc000f12000?, 0xc000f122c2?, 0x5?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc000392880 sp=0xc000392838 pc=0x2da9785
net.(*conn).Read(0xc0000bb868, {0xc000f12000?, 0x7f802021c948?, 0x7f806881f5c0?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc0003928c8 sp=0xc000392880 pc=0x2dbb825
crypto/tls.(*atLeastReader).Read(0xc000b105a0, {0xc000f12000?, 0x3239?, 0x2b551c5?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc000392910 sp=0xc0003928c8 pc=0x2e272bb
bytes.(*Buffer).ReadFrom(0xc000533ea8, {0x834e6c0, 0xc000b105a0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000392968 sp=0xc000392910 pc=0x2c22d18
crypto/tls.(*Conn).readFromUntil(0xc000533c08, {0x834f220, 0xc0000bb868}, 0xc0003929d0?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0003929a0 sp=0xc000392968 pc=0x2e2749e
crypto/tls.(*Conn).readRecordOrCCS(0xc000533c08, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000392c18 sp=0xc0003929a0 pc=0x2e2451b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000533c08, {0xc001591000, 0x1000, 0x3267100?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000392c80 sp=0xc000392c18 pc=0x2e2ae05
bufio.(*Reader).Read(0xc001455a40, {0xc0012bf004, 0x9, 0x32751ae?})
	/usr/lib/golang/src/bufio/bufio.go:245 +0x197 fp=0xc000392cb8 sp=0xc000392c80 pc=0x2cbe4d7
io.ReadAtLeast({0x834dce0, 0xc001455a40}, {0xc0012bf004, 0x9, 0x9}, 0x9)
	/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000392d00 sp=0xc000392cb8 pc=0x2bb2dce
io.ReadFull(...)
	/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0012bf004, 0x9, 0xc00000018b?}, {0x834dce0?, 0xc001455a40?})
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:250 +0x65 fp=0xc000392d50 sp=0xc000392d00 pc=0x325dec5
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0012befc0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:513 +0x6b fp=0xc000392dd8 sp=0xc000392d50 pc=0x325ea8b
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0012befc0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/frame.go:572 +0x18 fp=0xc000392e18 sp=0xc000392dd8 pc=0x325f1f8
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000392fa8)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2254 +0xca fp=0xc000392f60 sp=0xc000392e18 pc=0x32720ea
golang.org/x/net/http2.(*ClientConn).readLoop(0xc001049dc0)
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:2123 +0x52 fp=0xc000392fc8 sp=0xc000392f60 pc=0x3271472
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0x25 fp=0xc000392fe0 sp=0xc000392fc8 pc=0x326ac05
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000392fe8 sp=0xc000392fe0 pc=0x2b60c01
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 153
	/go/pkg/mod/golang.org/x/net@v0.52.0/http2/transport.go:876 +0xda5

rax    0xca
rbx    0x0
rcx    0x2b62a23
rdx    0x0
rdi    0xb57b7d0
rsi    0x80
rbp    0x7ffef1febfe0
rsp    0x7ffef1febf98
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xb5796e0
r15    0x3
rip    0x2b62a21
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349 +0x905 fp=0xc000d96960 sp=0xc000d95d58 pc=0x2fb2dc5
github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000b1ea88, {0x0, 0x0}, {0xb5a6520, 0x0, 0x0}, {0xc00007a094, 0x2a}, 0x0, {0xc000a93320, ...})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496 +0xa9b fp=0xc000d96e10 sp=0xc000d96960 pc=0x2fc68bb
github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007a094, 0x2a}, ...)
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68 +0x3c5 fp=0xc000d96eb8 sp=0xc000d96e10 pc=0x2fcdc25
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2ae7a85?, 0xc000610000?})
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/ginkgo/util.go:85 +0x2ba fp=0xc000d97b88 sp=0xc000d96eb8 pc=0x5394dda
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x83e90e8, 0xc000c146e0}, 0xc000cf0a80, 0x1)
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:596 +0xee fp=0xc000d97c30 sp=0xc000d97b88 pc=0x538b14e
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:237 +0x196 fp=0xc000d97fe0 sp=0xc000d97c30 pc=0x5386ed6
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d97fe8 sp=0xc000d97fe0 pc=0x2b60c01
created by github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230 +0x1e5

goroutine 96 gp=0xc000d58c40 m=nil [select]:
runtime.gopark(0xc000094ec8?, 0x2?, 0xc8?, 0xaa?, 0xc000094ebc?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000094d48 sp=0xc000094d28 pc=0x2b5848e
runtime.selectgo(0xc000094ec8, 0xc000094eb8, 0xc001486940?, 0x0, 0x33570dc?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000094e88 sp=0xc000094d48 pc=0x2b33277
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x83e90e8, 0xc000909ae0}, 0x79a871d?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000094ef8 sp=0xc000094e88 pc=0x3356836
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x841cda0, {0x83e90e8, 0xc000909ae0}, 0x0)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x7b2 fp=0xc000095240 sp=0xc000094ef8 pc=0x383e6f2
github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x83e9158?, 0xc0005293b0?}, {{0x83fef28?, 0xc0005fe200?}, 0xc0013f8ca8?}, 0xc000095748, {0xc0012c2960, 0x1e}, {0x7a1d023, 0x23}, ...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:694 +0x405 fp=0xc0000954b8 sp=0xc000095240 pc=0x4fc3965
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x7988ba7, 0x4}, ...}, ...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346 +0x497 fp=0xc000095818 sp=0xc0000954b8 pc=0x4fb7717
github.com/Azure/ARO-HCP/test/e2e.init.func24.2({0x7f0ed0192000, 0xc00086afc0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:82 +0x486 fp=0xc000095f58 sp=0xc000095818 pc=0x51854e6
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x83ff000?, 0xc00086afc0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e fp=0xc000095f88 sp=0xc000095f58 pc=0x2fb863e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000095fe0 sp=0xc000095f88 pc=0x2fccd5b
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 93
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

rax    0xfffffffffffffffc
rbx    0x0
rcx    0x2b62a23
rdx    0x0
rdi    0xb57d910
rsi    0x80
rbp    0x7f0ed2249cb8
rsp    0x7f0ed2249c70
r8     0x0
r9     0x0
r10    0x7f0ed2249ca8
r11    0x206
r12    0x7f0ed2249ca8
r13    0x0
r14    0xc000002c40
r15    0x0
rip    0x2b62a23
rflags 0x206
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/group.go:349 +0x905 fp=0xc000d96960 sp=0xc000d95d58 pc=0x2fb2dc5
github.com/onsi/ginkgo/v2/internal.(*Suite).runSpecs(0xc000b1ea88, {0x0, 0x0}, {0xb5a6520, 0x0, 0x0}, {0xc00007a094, 0x2a}, 0x0, {0xc000a93320, ...})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:496 +0xa9b fp=0xc000d96e10 sp=0xc000d96960 pc=0x2fc68bb
github.com/onsi/ginkgo/v2/internal.(*Suite).RunSpec(_, {_, _}, {_, _, _}, {_, _}, {0xc00007a094, 0x2a}, ...)
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite_patch.go:68 +0x3c5 fp=0xc000d96eb8 sp=0xc000d96e10 pc=0x2fcdc25
github.com/openshift-eng/openshift-tests-extension/pkg/ginkgo.BuildExtensionTestSpecsFromOpenShiftGinkgoSuite.func1.1({0x2ae7a85?, 0xc000610000?})
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/ginkgo/util.go:85 +0x2ba fp=0xc000d97b88 sp=0xc000d96eb8 pc=0x5394dda
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.runSpec({0x83e90e8, 0xc000c146e0}, 0xc000cf0a80, 0x1)
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:596 +0xee fp=0xc000d97c30 sp=0xc000d97b88 pc=0x538b14e
github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run.func2()
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:237 +0x196 fp=0xc000d97fe0 sp=0xc000d97c30 pc=0x5386ed6
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d97fe8 sp=0xc000d97fe0 pc=0x2b60c01
created by github.com/openshift-eng/openshift-tests-extension/pkg/extension/extensiontests.ExtensionTestSpecs.Run in goroutine 1
	/go/pkg/mod/github.com/openshift-eng/openshift-tests-extension@v0.0.0-20260127124016-0fed2b824818/pkg/extension/extensiontests/spec.go:230 +0x1e5

goroutine 96 gp=0xc000d58c40 m=nil [select]:
runtime.gopark(0xc000094ec8?, 0x2?, 0xc8?, 0xaa?, 0xc000094ebc?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000094d48 sp=0xc000094d28 pc=0x2b5848e
runtime.selectgo(0xc000094ec8, 0xc000094eb8, 0xc001486940?, 0x0, 0x33570dc?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000094e88 sp=0xc000094d48 pc=0x2b33277
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x83e90e8, 0xc000909ae0}, 0x79a871d?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/internal/shared/shared.go:36 +0x76 fp=0xc000094ef8 sp=0xc000094e88 pc=0x3356836
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x841cda0, {0x83e90e8, 0xc000909ae0}, 0x0)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.0/runtime/poller.go:284 +0x7b2 fp=0xc000095240 sp=0xc000094ef8 pc=0x383e6f2
github.com/Azure/ARO-HCP/test/util/framework.CreateHCPClusterAndWait({0x83e9158?, 0xc0005293b0?}, {{0x83fef28?, 0xc0005fe200?}, 0xc0013f8ca8?}, 0xc000095748, {0xc0012c2960, 0x1e}, {0x7a1d023, 0x23}, ...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:694 +0x405 fp=0xc0000954b8 sp=0xc000095240 pc=0x4fc3965
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CreateHCPClusterFromParam(_, {_, _}, {{_, _}, _}, {_, _}, {{0x7988ba7, 0x4}, ...}, ...)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/deployment_helper.go:346 +0x497 fp=0xc000095818 sp=0xc0000954b8 pc=0x4fb7717
github.com/Azure/ARO-HCP/test/e2e.init.func24.2({0x7f0ed0192000, 0xc00086afc0})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:82 +0x486 fp=0xc000095f58 sp=0xc000095818 pc=0x51854e6
github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x83ff000?, 0xc00086afc0?})
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 +0x3e fp=0xc000095f88 sp=0xc000095f58 pc=0x2fb863e
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc000095fe0 sp=0xc000095f88 pc=0x2fccd5b
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000095fe8 sp=0xc000095fe0 pc=0x2b60c01
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 93
	/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50

rax    0xfffffffffffffffc
rbx    0x0
rcx    0x2b62a23
rdx    0x0
rdi    0xb57d910
rsi    0x80
rbp    0x7f0ed2249cb8
rsp    0x7f0ed2249c70
r8     0x0
r9     0x0
r10    0x7f0ed2249ca8
r11    0x206
r12    0x7f0ed2249ca8
r13    0x0
r14    0xc000002c40
r15    0x0
rip    0x2b62a23
rflags 0x206
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserializaion Error: no output from command
Contributing tests (6)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to update node pool labels and taints2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
Affected runs (7)
Date (UTC)Associated PRProw job
n/a2038260698240258048
n/a2037413674255126528
n/a2037366470257676288
n/a2037065115634765824
n/a2037024456240533504
n/a2036937715248795648
n/a2036888953293901824
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
full failure pattern:
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e819.05%8 of 42 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsMar 22: 0 · Mar 23: 1 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 5 · Mar 28: 15 · Mar 29: 24none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapErrors | 0xc001508390>: 
    failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-nqc9bn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-nqc9bn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001392170>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:157]: Unexpected error:
    <*fmt.wrapErrors | 0xc001508390>: 
    failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-nqc9bn" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-nqc9bn\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001392170>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-nqc9bn",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:156]: Unexpected error:
    <*fmt.wrapErrors | 0xc00129fb00>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-2nlfhj" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-2nlfhj\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0007a6b60>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:156]: Unexpected error:
    <*fmt.wrapErrors | 0xc00129fb00>: 
    failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-2nlfhj" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-2nlfhj\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0007a6b60>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-2nlfhj",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
    <*fmt.wrapErrors | 0xc001211320>: 
    failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2m22rw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2m22rw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00092c660>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
    <*fmt.wrapErrors | 0xc001211320>: 
    failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-2m22rw" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw, error: context deadline exceeded
    {
        msg: "failed waiting for hcpCluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-2m22rw\" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00092c660>{
                s: "timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-2m22rw",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Contributing tests (18)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.195
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle5
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request4
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelControl plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.212
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to update node pool labels and taints2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelHCP Nodepools GPU instances creates and deletes vm type NC4asT4v3 in a single cluster2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelControl plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
Affected runs (8)
Date (UTC)Associated PRProw job
n/a2038260698240258048
n/a2038212316792623104
n/a2038107327839604736
n/a2037746786193379328
n/a2037699996773519360
n/a2037628298602745856
n/a2037557559711240192
n/a2036157936429961216
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; context ; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e37.14%3 of 42 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 5 · Mar 29: 2none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
    <*fmt.wrapError | 0xc0009e28a0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-vm728z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-vm728z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0008ed830>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["855"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260329T123708Z:c6998271-44f0-41ca-8da6-1f46b5fe6948",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: 29A874DFCC374E0391D1296600C0E257 Ref B: CH1AA2020617027 Ref C: 2026-03-29T12:37:07Z",
                    ],
                    "Date": [
                        "Sun, 29 Mar 2026 12:37:08 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Request-Id": [
                        "5bfee52a-0937-45f3-82af-d95d755a8729",
                    ],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/6a20fe18-7658-4e7c-bdbc-1e74dd3ff699",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "c6998271-44f0-41ca-8da6-1f46b5fe6948",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0009e27e0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "manageme...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:104]: Unexpected error:
    <*fmt.wrapError | 0xc0009e28a0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-vm728z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-vm728z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0008ed830>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Expires": ["-1"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["855"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260329T123708Z:c6998271-44f0-41ca-8da6-1f46b5fe6948",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "X-Msedge-Ref": [
                        "Ref A: 29A874DFCC374E0391D1296600C0E257 Ref B: CH1AA2020617027 Ref C: 2026-03-29T12:37:07Z",
                    ],
                    "Date": [
                        "Sun, 29 Mar 2026 12:37:08 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Request-Id": [
                        "5bfee52a-0937-45f3-82af-d95d755a8729",
                    ],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/6a20fe18-7658-4e7c-bdbc-1e74dd3ff699",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "c6998271-44f0-41ca-8da6-1f46b5fe6948",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Content-Type": ["application/json"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0009e27e0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                        Host: "manageme...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: Unexpected error:
    <*fmt.wrapError | 0xc0009aac00>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-79sq4q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-zskdm8/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-79sq4q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-zskdm8/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0010d7dd0>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Envoy-Upstream-Service-Time": ["560"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Request-Id": [
                        "8d6cae0d-8889-4205-aa31-fe38545dd223",
                    ],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/1101af73-6be3-42a3-a250-ff1e67521ad4",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Sun, 29 Mar 2026 12:37:09 GMT",
                    ],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "Expires": ["-1"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260329T123709Z:f22db835-02bb-4bfb-9e27-d21a5b4df8a1",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: 55288A5C640E4D0B9D2013D93A2A6472 Ref B: CHI261140510031 Ref C: 2026-03-29T12:37:08Z",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Correlation-Request-Id": [
                        "f22db835-02bb-4bfb-9e27-d21a5b4df8a1",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Content-Type": ["application/json"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0009aaac0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                   ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: Unexpected error:
    <*fmt.wrapError | 0xc0009aac00>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-79sq4q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-zskdm8/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-79sq4q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-zskdm8/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0010d7dd0>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Envoy-Upstream-Service-Time": ["560"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Request-Id": [
                        "8d6cae0d-8889-4205-aa31-fe38545dd223",
                    ],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/1101af73-6be3-42a3-a250-ff1e67521ad4",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Sun, 29 Mar 2026 12:37:09 GMT",
                    ],
                    "Pragma": ["no-cache"],
                    "Content-Length": ["107"],
                    "Expires": ["-1"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260329T123709Z:f22db835-02bb-4bfb-9e27-d21a5b4df8a1",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: 55288A5C640E4D0B9D2013D93A2A6472 Ref B: CHI261140510031 Ref C: 2026-03-29T12:37:08Z",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Correlation-Request-Id": [
                        "f22db835-02bb-4bfb-9e27-d21a5b4df8a1",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Content-Type": ["application/json"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0009aaac0>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                   ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: Unexpected error:
    <*fmt.wrapError | 0xc000886fe0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-k8s82w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gsxqhp/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-k8s82w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gsxqhp/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0006bc720>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Content-Type": ["application/json"],
                    "X-Msedge-Ref": [
                        "Ref A: 5C18A5E5865A447CAD257B2D17AC69BC Ref B: CHI261140511062 Ref C: 2026-03-28T05:01:46Z",
                    ],
                    "Expires": ["-1"],
                    "X-Ms-Request-Id": [
                        "b8001b45-128e-4419-b622-ebd59339ca7f",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["588"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260328T050147Z:2938cc51-e278-4bbf-8d54-92e75318a0ab",
                    ],
                    "Content-Length": ["107"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Sat, 28 Mar 2026 05:01:46 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/e2afbe11-ecaf-4b3c-9fb7-45c6d078ee48",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Correlation-Request-Id": [
                        "2938cc51-e278-4bbf-8d54-92e75318a0ab",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Pragma": ["no-cache"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000886e40>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                   ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:110]: Unexpected error:
    <*fmt.wrapError | 0xc000886fe0>: 
    failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-k8s82w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gsxqhp/requestAdminCredential
    --------------------------------------------------------------------------------
    RESPONSE 500: 500 Internal Server Error
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InternalServerError",
        "message": "Internal server error."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to start credential request: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-k8s82w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-gsxqhp/requestAdminCredential\n--------------------------------------------------------------------------------\nRESPONSE 500: 500 Internal Server Error\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Internal server error.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc0006bc720>{
            ErrorCode: "InternalServerError",
            StatusCode: 500,
            RawResponse: {
                Status: "500 Internal Server Error",
                StatusCode: 500,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Content-Type": ["application/json"],
                    "X-Msedge-Ref": [
                        "Ref A: 5C18A5E5865A447CAD257B2D17AC69BC Ref B: CHI261140511062 Ref C: 2026-03-28T05:01:46Z",
                    ],
                    "Expires": ["-1"],
                    "X-Ms-Request-Id": [
                        "b8001b45-128e-4419-b622-ebd59339ca7f",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["588"],
                    "X-Ms-Routing-Request-Id": [
                        "UKSOUTH:20260328T050147Z:2938cc51-e278-4bbf-8d54-92e75318a0ab",
                    ],
                    "Content-Length": ["107"],
                    "X-Ms-Failure-Cause": ["service"],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Date": [
                        "Sat, 28 Mar 2026 05:01:46 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "X-Ms-Error-Code": [
                        "InternalServerError",
                    ],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/e2afbe11-ecaf-4b3c-9fb7-45c6d078ee48",
                    ],
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["199"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2999"],
                    "X-Ms-Correlation-Request-Id": [
                        "2938cc51-e278-4bbf-8d54-92e75318a0ab",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Pragma": ["no-cache"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000886e40>{
                    s: [123, 10, 32, 32, 32, 32, 34, 101, 114, 114, 111, 114, 34, 58, 32, 123, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 99, 111, 100, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 83, 101, 114, 118, 101, 114, 69, 114, 114, 111, 114, 34, 44, 10, 32, 32, 32, 32, 32, 32, 32, 32, 34, 109, 101, 115, 115, 97, 103, 101, 34, 58, 32, 34, 73, 110, 116, 101, 114, 110, 97, 108, 32, 115, 101, 114, 118, 101, 114, 32, 101, 114, 114, 111, 114, 46, 34, 10, 32, 32, 32, 32, 125, 10, 125],
                    i: 0,
                },
                ContentLength: 107,
                TransferEncoding: nil,
                Close: false,
                Uncompressed: false,
                Trailer: nil,
                Request: {
                    Method: "POST",
                    URL: {
                        Scheme: "https",
                        Opaque: "",
                        User: nil,
                   ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelControl plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2038212316792623104
n/a2037746786193379328
n/a2037699996773519360
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e12.38%1 of 42 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 3 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:144]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ed0a50>: 
    failed to create NodePool gpu-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv, error: failed waiting for nodepool="gpu-np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-NC4asT4v3-n2zmdv" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool gpu-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv, error: failed waiting for nodepool=\"gpu-np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-n2zmdv\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000ae53f0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv",
            },
            <*fmt.wrapError | 0xc0008d4d00>{
                msg: "failed waiting for nodepool=\"gpu-np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-n2zmdv\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:144]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ed0a50>: 
    failed to create NodePool gpu-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv, error: failed waiting for nodepool="gpu-np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-NC4asT4v3-n2zmdv" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool gpu-np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv, error: failed waiting for nodepool=\"gpu-np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-n2zmdv\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000ae53f0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool gpu-np-1 in resource group rg-gpu-nodepool-NC4asT4v3-n2zmdv",
            },
            <*fmt.wrapError | 0xc0008d4d00>{
                msg: "failed waiting for nodepool=\"gpu-np-1\" for cluster \"cluster-gpu-np\" in resourcegroup=\"rg-gpu-nodepool-NC4asT4v3-n2zmdv\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:149]: Unexpected error:
    <*fmt.wrapErrors | 0xc000971080>: 
    failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94, error: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-jbg5qp" in resourcegroup="rg-np-version-upgrade-jbg5qp-5qgd94" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94, error: failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-jbg5qp\" in resourcegroup=\"rg-np-version-upgrade-jbg5qp-5qgd94\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009509c0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94",
            },
            <*fmt.wrapError | 0xc0006cafc0>{
                msg: "failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-jbg5qp\" in resourcegroup=\"rg-np-version-upgrade-jbg5qp-5qgd94\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:149]: Unexpected error:
    <*fmt.wrapErrors | 0xc000971080>: 
    failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94, error: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-jbg5qp" in resourcegroup="rg-np-version-upgrade-jbg5qp-5qgd94" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94, error: failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-jbg5qp\" in resourcegroup=\"rg-np-version-upgrade-jbg5qp-5qgd94\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0009509c0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-jbg5qp-5qgd94",
            },
            <*fmt.wrapError | 0xc0006cafc0>{
                msg: "failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-jbg5qp\" in resourcegroup=\"rg-np-version-upgrade-jbg5qp-5qgd94\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:149]: Unexpected error:
    <*fmt.wrapErrors | 0xc00141ce40>: 
    failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4, error: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-vx4kj8" in resourcegroup="rg-np-version-upgrade-vx4kj8-59lpv4" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4, error: failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-vx4kj8\" in resourcegroup=\"rg-np-version-upgrade-vx4kj8-59lpv4\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0015989a0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4",
            },
            <*fmt.wrapError | 0xc001094800>{
                msg: "failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-vx4kj8\" in resourcegroup=\"rg-np-version-upgrade-vx4kj8-59lpv4\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:149]: Unexpected error:
    <*fmt.wrapErrors | 0xc00141ce40>: 
    failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4, error: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-vx4kj8" in resourcegroup="rg-np-version-upgrade-vx4kj8-59lpv4" to finish creating: context deadline exceeded
    {
        msg: "failed to create NodePool npupgrade-4-20, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4, error: failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-vx4kj8\" in resourcegroup=\"rg-np-version-upgrade-vx4kj8-59lpv4\" to finish creating: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0015989a0>{
                s: "timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-vx4kj8-59lpv4",
            },
            <*fmt.wrapError | 0xc001094800>{
                msg: "failed waiting for nodepool=\"npupgrade-4-20\" for cluster \"np-version-upgrade-cluster-vx4kj8\" in resourcegroup=\"rg-np-version-upgrade-vx4kj8-59lpv4\" to finish creating: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        ],
    }
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should upgrade a nodepool from 4.20.z to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should upgrade a nodepool from 4.20.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelHCP Nodepools GPU instances creates and deletes vm type NC4asT4v3 in a single cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2037303065828134912
ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscri...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.38%1 of 42 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 2 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc000805a20>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-55ctkl" in resourcegroup="customer-rg-97sgrb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-55ctkl\" in resourcegroup=\"customer-rg-97sgrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000805a00>{
            msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-55ctkl\" in resourcegroup=\"customer-rg-97sgrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001129e30>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc000805a20>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-55ctkl" in resourcegroup="customer-rg-97sgrb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-55ctkl\" in resourcegroup=\"customer-rg-97sgrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000805a00>{
            msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-55ctkl\" in resourcegroup=\"customer-rg-97sgrb\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-97sgrb/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-55ctkl/operationStatuses/08584270498462875647\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-97sgrb/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001129e30>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc000899900>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-rc7l8v" in resourcegroup="rg-cluster-nsg-subnet-reuse-5rm9ww" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-rc7l8v\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-5rm9ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008998e0>{
            msg: "failed waiting for deployment \"customer-infra-basic-cluster-rc7l8v\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-5rm9ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000e559e0>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 2...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc000899900>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-rc7l8v" in resourcegroup="rg-cluster-nsg-subnet-reuse-5rm9ww" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "BadRequest",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n    \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n    \"details\": []\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-rc7l8v\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-5rm9ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008998e0>{
            msg: "failed waiting for deployment \"customer-infra-basic-cluster-rc7l8v\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-5rm9ww\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-rc7l8v/operationStatuses/08584270500153299231\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n    \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-5rm9ww/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n    \\\"details\\\": []\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000e559e0>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 2...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2037264652152344576
ERROR CODE: InternalServerError; detail message SAL validation failed with status code: 403; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message SAL validation failed with status code: 403; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.38%1 of 42 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 2 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
    <*fmt.wrapError | 0xc0008a5220>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-zvtsqs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0",
      "name": "424c0c62-70a3-4351-ac27-2c33c7bb83b0",
      "status": "Failed",
      "startTime": "2026-03-26T20:30:39.925830532Z",
      "endTime": "2026-03-26T20:30:48.906575258Z",
      "error": {
        "code": "InternalServerError",
        "message": "SAL validation failed with status code: 403"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zvtsqs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n  \"name\": \"424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-26T20:30:39.925830532Z\",\n  \"endTime\": \"2026-03-26T20:30:48.906575258Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"SAL validation failed with status code: 403\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008a5200>{
            msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zvtsqs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n  \"name\": \"424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-26T20:30:39.925830532Z\",\n  \"endTime\": \"2026-03-26T20:30:48.906575258Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"SAL validation failed with status code: 403\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011583c0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 5A0DC58691384AA18C1C9F410C4E5748 Ref B: CHI261140508054 Ref C: 2026-03-26T20:30:50Z",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Correlation-Request-Id": [
                            "9ebf9896-a696-4ba0-a35b-c86547212be5",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "d7646fbd-d48e-43c2-9674-7365e0a2a6fa",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadaeast/b22816e7-8010-49c2-b249-f2327a9fe3fb",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
    <*fmt.wrapError | 0xc0008a5220>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-zvtsqs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0",
      "name": "424c0c62-70a3-4351-ac27-2c33c7bb83b0",
      "status": "Failed",
      "startTime": "2026-03-26T20:30:39.925830532Z",
      "endTime": "2026-03-26T20:30:48.906575258Z",
      "error": {
        "code": "InternalServerError",
        "message": "SAL validation failed with status code: 403"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zvtsqs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n  \"name\": \"424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-26T20:30:39.925830532Z\",\n  \"endTime\": \"2026-03-26T20:30:48.906575258Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"SAL validation failed with status code: 403\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008a5200>{
            msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-zvtsqs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n  \"name\": \"424c0c62-70a3-4351-ac27-2c33c7bb83b0\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-26T20:30:39.925830532Z\",\n  \"endTime\": \"2026-03-26T20:30:48.906575258Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"SAL validation failed with status code: 403\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011583c0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 5A0DC58691384AA18C1C9F410C4E5748 Ref B: CHI261140508054 Ref C: 2026-03-26T20:30:50Z",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Correlation-Request-Id": [
                            "9ebf9896-a696-4ba0-a35b-c86547212be5",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "d7646fbd-d48e-43c2-9674-7365e0a2a6fa",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadaeast/b22816e7-8010-49c2-b249-f2327a9fe3fb",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
    <*fmt.wrapError | 0xc000aafca0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-qjbk7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9",
      "name": "34cd3f19-97fd-4358-957a-9ecd7e60d0f9",
      "status": "Failed",
      "startTime": "2026-03-26T20:30:28.750332994Z",
      "endTime": "2026-03-26T20:30:48.509300921Z",
      "error": {
        "code": "InternalServerError",
        "message": "SAL validation failed with status code: 403"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-qjbk7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9\",\n  \"name\": \"34cd3f19-97fd-4358-957a-9ecd7e60d0f9\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-26T20:30:28.750332994Z\",\n  \"endTime\": \"2026-03-26T20:30:48.509300921Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"SAL validation failed with status code: 403\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000e36660>{
            ErrorCode: "InternalServerError",
            StatusCode: 200,
            RawResponse: {
                Status: "200 OK",
                StatusCode: 200,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Expires": ["-1"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Content-Length": ["481"],
                    "X-Envoy-Upstream-Service-Time": ["18"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadacentral/d2b1b180-95e6-469a-a3cb-7097078b6027",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Ms-Request-Id": [
                        "886185d4-7470-4791-ba1e-90e0db1c201d",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "74b1db3f-fa85-456f-8b00-4f0f4ca09c9e",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "CANADACENTRAL:20260326T203051Z:74b1db3f-fa85-456f-8b00-4f0f4ca09c9e",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Content-Type": ["application/json"],
                    "X-Msedge-Ref": [
                        "Ref A: 72447DB8C47A4BA58F26EB6D90519D0E Ref B: CHI261140501042 Ref C: 2026-03-26T20:30:51Z",
                    ],
                    "Date": [
                        "Thu, 26 Mar 2026 20:30:51 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000aafb80>{
                    s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 101, 98, 99, 50, 45, 52, 49, 53, 54, 45, 57, 100, 57, 49, 45, 99, 52, 99, 55, 56, 49, 100, 101, 55, 101, 53, 52, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 117, 107, 115, 111, 117, 116, 104, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 51, 52, 99, 100, 51, 102, 49, 57, 45, 57, 55, 102, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
    <*fmt.wrapError | 0xc000aafca0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-qjbk7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9",
      "name": "34cd3f19-97fd-4358-957a-9ecd7e60d0f9",
      "status": "Failed",
      "startTime": "2026-03-26T20:30:28.750332994Z",
      "endTime": "2026-03-26T20:30:48.509300921Z",
      "error": {
        "code": "InternalServerError",
        "message": "SAL validation failed with status code: 403"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-qjbk7k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34cd3f19-97fd-4358-957a-9ecd7e60d0f9\",\n  \"name\": \"34cd3f19-97fd-4358-957a-9ecd7e60d0f9\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-26T20:30:28.750332994Z\",\n  \"endTime\": \"2026-03-26T20:30:48.509300921Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"SAL validation failed with status code: 403\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000e36660>{
            ErrorCode: "InternalServerError",
            StatusCode: 200,
            RawResponse: {
                Status: "200 OK",
                StatusCode: 200,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Expires": ["-1"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                    "X-Cache": ["CONFIG_NOCACHE"],
                    "Content-Length": ["481"],
                    "X-Envoy-Upstream-Service-Time": ["18"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/canadacentral/d2b1b180-95e6-469a-a3cb-7097078b6027",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Ms-Request-Id": [
                        "886185d4-7470-4791-ba1e-90e0db1c201d",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "74b1db3f-fa85-456f-8b00-4f0f4ca09c9e",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "CANADACENTRAL:20260326T203051Z:74b1db3f-fa85-456f-8b00-4f0f4ca09c9e",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "Content-Type": ["application/json"],
                    "X-Msedge-Ref": [
                        "Ref A: 72447DB8C47A4BA58F26EB6D90519D0E Ref B: CHI261140501042 Ref C: 2026-03-26T20:30:51Z",
                    ],
                    "Date": [
                        "Thu, 26 Mar 2026 20:30:51 GMT",
                    ],
                },
                Body: <*exported.nopClosingBytesReader | 0xc000aafb80>{
                    s: [123, 10, 32, 32, 32, 32, 34, 105, 100, 34, 58, 32, 34, 47, 115, 117, 98, 115, 99, 114, 105, 112, 116, 105, 111, 110, 115, 47, 54, 52, 102, 48, 54, 49, 57, 102, 45, 101, 98, 99, 50, 45, 52, 49, 53, 54, 45, 57, 100, 57, 49, 45, 99, 52, 99, 55, 56, 49, 100, 101, 55, 101, 53, 52, 47, 112, 114, 111, 118, 105, 100, 101, 114, 115, 47, 77, 105, 99, 114, 111, 115, 111, 102, 116, 46, 82, 101, 100, 72, 97, 116, 79, 112, 101, 110, 83, 104, 105, 102, 116, 47, 108, 111, 99, 97, 116, 105, 111, 110, 115, 47, 117, 107, 115, 111, 117, 116, 104, 47, 104, 99, 112, 79, 112, 101, 114, 97, 116, 105, 111, 110, 83, 116, 97, 116, 117, 115, 101, 115, 47, 51, 52, 99, 100, 51, 102, 49, 57, 45, 57, 55, 102, ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2037264652152344576
ERROR CODE: Conflict; detail message Resource is already deleting; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: Conflict; detail message Resource is already deleting; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e12.38%1 of 42 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
    <*errors.joinError | 0xc00141cd38>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Resource is already deleting",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*fmt.wrapError | 0xc0008bb5e0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Resource is already deleting\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*fmt.wrapError | 0xc0008bb5c0>{
                    msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Resource is already deleting\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                    err: <*exported.ResponseError | 0xc00178e6c0>{
                        ErrorCode: "Conflict",
                        StatusCode: 409,
                        RawResponse: {
                            Status: "409 Conflict",
                            StatusCode: 409,
                            Proto: "HTTP/2.0",
                            ProtoMajor: 2,
                            ProtoMinor: 0,
                            Header: {
                                "Pragma": ["no-cache"],
                                "Expires": ["-1"],
                                "X-Ms-Request-Id": [
                                    "28ede0cd-2936-4f2f-8253-cb605ecc9170",
                                ],
                                "X-Ms-Ratelimit-Remaining-Subscription-Deletes": ["199"],
                                "X-Ms-Routing-Request-Id": [
                                    "UKSOUTH:20260325T152452Z:5c77f5c4-ef8b-4b65-ad8a-d8b321c39862",
                                ],
                                "Strict-Transport-Security": [
                                    "max-age=31536000; includeSubDomains",
                                ],
                                "Content-Length": ["295"],
                                "Content-Type": ["application/json"],
                                "X-Ms-Correlation-Request-Id": [
                                    "5c77f5c4-ef8b-4b65-ad8a-d8b321c39862",
                                ],
                                "Cache-Control": ["no-cache"],
                                "X-Ms-Error-Code": ["Conflict"],
                                "X-Envoy-Upstream-Service-Time": ["1026"],
                                "X-Ms-Operation-Identifier": [
                                    "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/5c5c915c-65f2-46d6-a279-f918cf82b3b5",
                                ],
                                "X-Ms-Ratelimit-Remaining-Subscription-Global-Del...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
    <*errors.joinError | 0xc00141cd38>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Resource is already deleting",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        errs: [
            <*fmt.wrapError | 0xc0008bb5e0>{
                msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Resource is already deleting\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                err: <*fmt.wrapError | 0xc0008bb5c0>{
                    msg: "at least one hcp cluster failed to delete: DELETE https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"Conflict\",\n    \"message\": \"Resource is already deleting\",\n    \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cluster-listing-r8tknz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/list-test-cluster-lkzjxd\"\n  }\n}\n--------------------------------------------------------------------------------\n",
                    err: <*exported.ResponseError | 0xc00178e6c0>{
                        ErrorCode: "Conflict",
                        StatusCode: 409,
                        RawResponse: {
                            Status: "409 Conflict",
                            StatusCode: 409,
                            Proto: "HTTP/2.0",
                            ProtoMajor: 2,
                            ProtoMinor: 0,
                            Header: {
                                "Pragma": ["no-cache"],
                                "Expires": ["-1"],
                                "X-Ms-Request-Id": [
                                    "28ede0cd-2936-4f2f-8253-cb605ecc9170",
                                ],
                                "X-Ms-Ratelimit-Remaining-Subscription-Deletes": ["199"],
                                "X-Ms-Routing-Request-Id": [
                                    "UKSOUTH:20260325T152452Z:5c77f5c4-ef8b-4b65-ad8a-d8b321c39862",
                                ],
                                "Strict-Transport-Security": [
                                    "max-age=31536000; includeSubDomains",
                                ],
                                "Content-Length": ["295"],
                                "Content-Type": ["application/json"],
                                "X-Ms-Correlation-Request-Id": [
                                    "5c77f5c4-ef8b-4b65-ad8a-d8b321c39862",
                                ],
                                "Cache-Control": ["no-cache"],
                                "X-Ms-Error-Code": ["Conflict"],
                                "X-Envoy-Upstream-Service-Time": ["1026"],
                                "X-Ms-Operation-Identifier": [
                                    "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=c209f8df-52ae-48fb-98ea-380f58b04652/uksouth/5c5c915c-65f2-46d6-a279-f918cf82b3b5",
                                ],
                                "X-Ms-Ratelimit-Remaining-Subscription-Global-Del...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to list HCP clusters without node pools at the resource group level1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2036825136069349376
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.38%1 of 42 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 1 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f66e40>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-ct8x6p" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-ct8x6p\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001606080>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:183]: Unexpected error:
    <*fmt.wrapErrors | 0xc000f66e40>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-ct8x6p" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p, error: context deadline exceeded
    {
        msg: "failed waiting for nodepool=\"np-1\" in cluster=\"cluster-np-labels-taints\" resourcegroup=\"rg-np-labels-taints-ct8x6p\" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001606080>{
                s: "timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-ct8x6p",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2037303065828134912
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq
full failure pattern:
found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.38%1 of 42 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 1 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
    <*errors.joinError | 0xc000d9d878>: 
    found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq
    {
        errs: [
            <*errors.errorString | 0xc000718bc0>{
                s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq",
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
    <*errors.joinError | 0xc000d9d878>: 
    found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq
    {
        errs: [
            <*errors.errorString | 0xc000718bc0>{
                s: "found 1 managed resource groups left behind HCP clusters in rg-cluster-nsg-subnet-reuse-fdfcnq",
            },
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2036825136069349376
operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:
full failure pattern:
operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e12.38%1 of 42 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsMar 22: 0 · Mar 23: 0 · Mar 24: 1 · Mar 25: 0 · Mar 26: 0 · Mar 27: 0 · Mar 28: 0 · Mar 29: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error:
    <*errors.errorString | 0xc000a33960>: 
    operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:
      &armredhatopenshifthcp.HcpOpenShiftCluster{
      	Location: &"uksouth",
      	Identity: &{Type: &"UserAssigned", UserAssignedIdentities: {"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, ...}},
      	Properties: &armredhatopenshifthcp.HcpOpenShiftClusterProperties{
      		... // 7 identical fields
      		NodeDrainTimeoutMinutes: nil,
      		Version:                 &{ChannelGroup: &"stable", ID: &"4.20"},
    - 		Console: &armredhatopenshifthcp.ConsoleProfile{
    - 			URL: &"https://console-openshift-console.apps.aro.u9t3y3i9f1v2d7g.8p8o.uksouth.aroapp-hcp.azure-test.net",
    - 		},
    + 		Console:           nil,
      		ProvisioningState: &"Succeeded",
      	},
      	Tags: {"aro-hcp.experimental.cluster.size-override": &"Minimal"},
      	ID:   &"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGrou"...,
      	... // 1 ignored and 2 identical fields
      }
    
    {
        s: "operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:\n\u00a0\u00a0&armredhatopenshifthcp.HcpOpenShiftCluster{\n\u00a0\u00a0\tLocation: &\"uksouth\",\n\u00a0\u00a0\tIdentity: &{Type: &\"UserAssigned\", UserAssignedIdentities: {\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, ...}},\n\u00a0\u00a0\tProperties: &armredhatopenshifthcp.HcpOpenShiftClusterProperties{\n\u00a0\u00a0\t\t... // 7 identical fields\n\u00a0\u00a0\t\tNodeDrainTimeoutMinutes: nil,\n\u00a0\u00a0\t\tVersion:                 &{ChannelGroup: &\"stable\", ID: &\"4.20\"},\n-\u00a0\t\tConsole: &armredhatopenshifthcp.ConsoleProfile{\n-\u00a0\t\t\tURL: &\"https://console-openshift-console.apps.aro.u9t3y3i9f1v2d7g.8p8o.uksouth.aroapp-hcp.azure-test.net\",\n-\u00a0\t\t},\n+\u00a0\t\tConsole:           nil,\n\u00a0\u00a0\t\tProvisioningState: &\"Succeeded\",\n\u00a0\u00a0\t},\n\u00a0\u00a0\tTags: {\"aro-hcp.experimental.cluster.size-override\": &\"Minimal\"},\n\u00a0\u00a0\tID:   &\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGrou\"...,\n\u00a0\u00a0\t... // 1 ignored and 2 identical fields\n\u00a0\u00a0}\n",
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:141]: Unexpected error:
    <*errors.errorString | 0xc000a33960>: 
    operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:
      &armredhatopenshifthcp.HcpOpenShiftCluster{
      	Location: &"uksouth",
      	Identity: &{Type: &"UserAssigned", UserAssignedIdentities: {"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane": &{ClientID: &"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a", PrincipalID: &"a5c120f7-50c3-4cf6-84b4-57517f60630d"}, ...}},
      	Properties: &armredhatopenshifthcp.HcpOpenShiftClusterProperties{
      		... // 7 identical fields
      		NodeDrainTimeoutMinutes: nil,
      		Version:                 &{ChannelGroup: &"stable", ID: &"4.20"},
    - 		Console: &armredhatopenshifthcp.ConsoleProfile{
    - 			URL: &"https://console-openshift-console.apps.aro.u9t3y3i9f1v2d7g.8p8o.uksouth.aroapp-hcp.azure-test.net",
    - 		},
    + 		Console:           nil,
      		ProvisioningState: &"Succeeded",
      	},
      	Tags: {"aro-hcp.experimental.cluster.size-override": &"Minimal"},
      	ID:   &"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGrou"...,
      	... // 1 ignored and 2 identical fields
      }
    
    {
        s: "operation result model did not match expected model for type *armredhatopenshifthcp.HcpOpenShiftCluster:\n\u00a0\u00a0&armredhatopenshifthcp.HcpOpenShiftCluster{\n\u00a0\u00a0\tLocation: &\"uksouth\",\n\u00a0\u00a0\tIdentity: &{Type: &\"UserAssigned\", UserAssignedIdentities: {\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-int-136/providers/Microsoft.ManagedIdentity/userAssignedIdentities/control-plane\": &{ClientID: &\"f2e4769e-d3c2-498d-92b9-3e6d24cd2d7a\", PrincipalID: &\"a5c120f7-50c3-4cf6-84b4-57517f60630d\"}, ...}},\n\u00a0\u00a0\tProperties: &armredhatopenshifthcp.HcpOpenShiftClusterProperties{\n\u00a0\u00a0\t\t... // 7 identical fields\n\u00a0\u00a0\t\tNodeDrainTimeoutMinutes: nil,\n\u00a0\u00a0\t\tVersion:                 &{ChannelGroup: &\"stable\", ID: &\"4.20\"},\n-\u00a0\t\tConsole: &armredhatopenshifthcp.ConsoleProfile{\n-\u00a0\t\t\tURL: &\"https://console-openshift-console.apps.aro.u9t3y3i9f1v2d7g.8p8o.uksouth.aroapp-hcp.azure-test.net\",\n-\u00a0\t\t},\n+\u00a0\t\tConsole:           nil,\n\u00a0\u00a0\t\tProvisioningState: &\"Succeeded\",\n\u00a0\u00a0\t},\n\u00a0\u00a0\tTags: {\"aro-hcp.experimental.cluster.size-override\": &\"Minimal\"},\n\u00a0\u00a0\tID:   &\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGrou\"...,\n\u00a0\u00a0\t... // 1 ignored and 2 identical fields\n\u00a0\u00a0}\n",
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-integration-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2036284853137182720