<
Weekly: Jun 15 - Jun 21
> Reset
View JSON API

Environment: PROD

Failure patterns: 34 · Failures matched: 221 · Total runs: 68 · Runs affected: 40

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
e2e2638.24%26 of 68 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsJun 14: 0 · Jun 15: 4 · Jun 16: 6 · Jun 17: 15 · Jun 18: 3 · Jun 19: 22 · Jun 20: 8 · Jun 21: 6none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-tks2fl" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00138c300>: 
    failed to create HCP cluster cluster-zstream-4-22-tks2fl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-tks2fl in resource group rg-zstream-upgrade-4-22-dff7ch, error: failed waiting for cluster="cluster-zstream-4-22-tks2fl" in resourcegroup="rg-zstream-upgrade-4-22-dff7ch" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-tks2fl in resource group rg-zstream-upgrade-4-22-dff7ch, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-tks2fl" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00138c300>: 
    failed to create HCP cluster cluster-zstream-4-22-tks2fl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-tks2fl in resource group rg-zstream-upgrade-4-22-dff7ch, error: failed waiting for cluster="cluster-zstream-4-22-tks2fl" in resourcegroup="rg-zstream-upgrade-4-22-dff7ch" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-tks2fl in resource group rg-zstream-upgrade-4-22-dff7ch, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-prpz8r" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc000ff0660>: 
    failed to create HCP cluster cluster-zstream-4-22-prpz8r, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-prpz8r in resource group rg-zstream-upgrade-4-22-xncgh2, error: failed waiting for cluster="cluster-zstream-4-22-prpz8r" in resourcegroup="rg-zstream-upgrade-4-22-xncgh2" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-prpz8r in resource group rg-zstream-upgrade-4-22-xncgh2, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-prpz8r" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc000ff0660>: 
    failed to create HCP cluster cluster-zstream-4-22-prpz8r, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-prpz8r in resource group rg-zstream-upgrade-4-22-xncgh2, error: failed waiting for cluster="cluster-zstream-4-22-prpz8r" in resourcegroup="rg-zstream-upgrade-4-22-xncgh2" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-prpz8r in resource group rg-zstream-upgrade-4-22-xncgh2, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapErrors | 0xc0012eec00>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-x7v479, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-x7v479" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-x7v479, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapErrors | 0xc0012eec00>: 
    failed to create HCP cluster idms-e2e-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-x7v479, error: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-x7v479" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster idms-e2e-hcp-cluster in resource group idms-x7v479, error: context deadline exceeded
    ...
occurred
Contributing tests (39)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.227
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.03
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.03
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.03
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.232
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelMISE Routing routes to the correct frontend based on version header default route returns no version header2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyMISE Routing routes to the correct frontend based on version header default route returns no version header1
Affected runs (26)
Date (UTC)Associated PRProw job
n/a2068831445857603584
n/a2068591174670618624
n/a2068514352226373632
n/a2068492780056875008
n/a2068469052392607744
n/a2068465010476060672
n/a2068443878926585856
n/a2068402345783332864
n/a2068216057461477376
n/a2068215937068175360
n/a2068082178301890560
n/a2068063774274752512
n/a2068063035146113024
n/a2068004614032068608
n/a2067949798622760960
n/a2067929498837323776
n/a2067898279315640320
n/a2067643701898776576
n/a2067524854185201664
n/a2067472693011156992
n/a2067198813889630208
n/a2067064761694031872
n/a2067019458672594944
n/a2066702365632761856
n/a2066657066361032704
n/a2066339954090315776
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 8 days
e2e1826.47%18 of 68 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 8 daysShow trend detailsJun 14: 8 · Jun 15: 9 · Jun 16: 7 · Jun 17: 9 · Jun 18: 3 · Jun 19: 7 · Jun 20: 1 · Jun 21: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:212]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapErrors | 0xc001215e90>: 
    failed to create NodePool autoscale-noaz, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool autoscale-noaz in resource group np-autoscaling-5rk4ks, error: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-5rk4ks" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:212]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapErrors | 0xc001215e90>: 
    failed to create NodePool autoscale-noaz, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool autoscale-noaz in resource group np-autoscaling-5rk4ks, error: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-5rk4ks" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:137]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000957830>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-dtvsk2, error: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dtvsk2" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:137]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000957830>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-dtvsk2, error: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dtvsk2" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:126]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc001255950>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-5rpxgq, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5rpxgq" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:126]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc001255950>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-5rpxgq, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5rpxgq" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (21)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling5
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.203
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.213
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.202
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.212
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (18)
Date (UTC)Associated PRProw job
n/a2068641277968125952
n/a2068591174670618624
n/a2068520879767162880
n/a2068402345783332864
n/a2068004614032068608
n/a2067949798622760960
n/a2067929498837323776
n/a2067889808788688896
n/a2067779867528663040
n/a2067744257157894144
n/a2067643701898776576
n/a2067598465302532096
n/a2067198813889630208
n/a2067064761694031872
n/a2067019458672594944
n/a2066702365632761856
n/a2066657066361032704
n/a2066339954090315776
Interrupted by User
full failure pattern:
Interrupted by User
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
e2e811.76%8 of 68 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 1 · Jun 18: 0 · Jun 19: 3 · Jun 20: 2 · Jun 21: 4none
Full failure examples (1)
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
Contributing tests (7)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.202
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (8)
Date (UTC)Associated PRProw job
n/a2068641277968125952
n/a2068591174670618624
n/a2068432057893130240
n/a2068402345783332864
n/a2068082178301890560
n/a2068004614032068608
n/a2067929498837323776
n/a2067198813889630208
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
e2e811.76%8 of 68 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsJun 14: 0 · Jun 15: 1 · Jun 16: 2 · Jun 17: 0 · Jun 18: 1 · Jun 19: 1 · Jun 20: 2 · Jun 21: 1none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc001090000>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc001090000>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc00160ea98>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc00160ea98>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00106a780>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-26pckg, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-26pckg" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00106a780>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-26pckg, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-26pckg" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (8)
Date (UTC)Associated PRProw job
n/a2068591174670618624
n/a2068402345783332864
n/a2068399947908124672
n/a2067949798622760960
n/a2067641305726455808
n/a2067019458672594944
n/a2066702365632761856
n/a2066339954090315776
Deserialization Error: no output from command
full failure pattern:
Deserialization Error: no output from command
Signal: NoiseSignal: Noise — source deserialization/no-output error — New failure pattern — no prior history
e2e57.35%5 of 68 job runs affectedNoiseSignal: Noise — source deserialization/no-output error — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 2 · Jun 21: 3none
Full failure examples (3)
fp=0xc0017ad958 sp=0xc0017ad910 pc=0x31eef1b
bytes.(*Buffer).ReadFrom(0xc000c582a8, {0x9220320, 0xc0016f5068})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc0017ad9b0 sp=0xc0017ad958 pc=0x2fef578
crypto/tls.(*Conn).readFromUntil(0xc000c58008, {0x9221e00, 0xc000d99020}, 0xc0017ada18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0017ad9e8 sp=0xc0017ad9b0 pc=0x31ef0fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000c58008, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc0017adc60 sp=0xc0017ad9e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000c58008, {0xc0018a0000, 0x1000, 0xc0009a9080?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc0017adcc8 sp=0xc0017adc60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc00184e6c0, {0xc0018a0000?, 0x32dcb05?, 0x921acc0?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc0017add28 sp=0xc0017adcc8 pc=0x32e4407
bufio.(*Reader).fill(0xc0012731a0)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc0017add60 sp=0xc0017add28 pc=0x3084943
bufio.(*Reader).Peek(0xc0012731a0, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc0017add80 sp=0xc0017add60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc00184e6c0)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc0017adfc8 sp=0xc0017add80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc0017adfe0 sp=0xc0017adfc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0017adfe8 sp=0xc0017adfe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 206
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 249 gp=0xc000683c00 m=nil [select]:
runtime.gopark(0xc001247768?, 0x2?, 0x40?, 0x8?, 0xc00124775c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0012475e8 sp=0xc0012475c8 pc=0x2f1e50e
runtime.selectgo(0xc001247768, 0xc001247758, 0xc000fb27a0?, 0x0, 0x36ecd5c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc001247728 sp=0xc0012475e8 pc=0x2ef92f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x92b8658, 0xc0018b60e0}, 0x8704ddd?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc001247798 sp=0xc001247728 pc=0x36ec4b6
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x930a440, {0x92b8658, 0xc0018b60e0}, 0x2f1baf3)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc001247a98 sp=0xc001247798 pc=0x3ded372
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x92b8658, 0xc0018b60e0}, 0xc0016f51a0, {0xc000c1d728, 0x18}, {0xc00181f620, 0x1d})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:240 +0x365 fp=0xc001247c20 sp=0xc001247a98 pc=0x6c2ef65
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x92b8658?, 0xc0018b60e0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x72 fp=0xc001247cc8 sp=0xc001247c20 pc=0x6c2e992
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x92b8658?, 0xc0018b60e0?}, 0xc0018b60f0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc001247d20 sp=0xc001247cc8 pc=0x369fa82
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x92b8658, 0xc0018b60e0}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc001247e68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc001247d98 sp=0xc001247d20 pc=0x369db2e
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x92b85e8, 0xc000b615e0}, 0xc0016f51a0, {0xc000c1d728, 0x18}, {0xc00181f620, 0x1d}, 0x274a48a7800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:196 +0x285 fp=0xc001247ef0 sp=0xc001247d98 pc=0x6c2e705
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:522 +0x7e fp=0xc001247f78 sp=0xc001247ef0 pc=0x6c325fe
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc001247fe0 sp=0xc001247f78 pc=0x4a4be30
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001247fe8 sp=0xc001247fe0 pc=0x2f26cc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 248
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x2f28ae3
rdx    0x0
rdi    0xc93a4a0
rsi    0x80
rbp    0xc0000b5728
rsp    0xc0000b56e0
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xc000582540
r15    0x7
rip    0x2f28ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
fp=0xc0017ad958 sp=0xc0017ad910 pc=0x31eef1b
bytes.(*Buffer).ReadFrom(0xc000c582a8, {0x9220320, 0xc0016f5068})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc0017ad9b0 sp=0xc0017ad958 pc=0x2fef578
crypto/tls.(*Conn).readFromUntil(0xc000c58008, {0x9221e00, 0xc000d99020}, 0xc0017ada18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0017ad9e8 sp=0xc0017ad9b0 pc=0x31ef0fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000c58008, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc0017adc60 sp=0xc0017ad9e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000c58008, {0xc0018a0000, 0x1000, 0xc0009a9080?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc0017adcc8 sp=0xc0017adc60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc00184e6c0, {0xc0018a0000?, 0x32dcb05?, 0x921acc0?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc0017add28 sp=0xc0017adcc8 pc=0x32e4407
bufio.(*Reader).fill(0xc0012731a0)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc0017add60 sp=0xc0017add28 pc=0x3084943
bufio.(*Reader).Peek(0xc0012731a0, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc0017add80 sp=0xc0017add60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc00184e6c0)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc0017adfc8 sp=0xc0017add80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc0017adfe0 sp=0xc0017adfc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0017adfe8 sp=0xc0017adfe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 206
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 249 gp=0xc000683c00 m=nil [select]:
runtime.gopark(0xc001247768?, 0x2?, 0x40?, 0x8?, 0xc00124775c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0012475e8 sp=0xc0012475c8 pc=0x2f1e50e
runtime.selectgo(0xc001247768, 0xc001247758, 0xc000fb27a0?, 0x0, 0x36ecd5c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc001247728 sp=0xc0012475e8 pc=0x2ef92f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x92b8658, 0xc0018b60e0}, 0x8704ddd?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc001247798 sp=0xc001247728 pc=0x36ec4b6
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x930a440, {0x92b8658, 0xc0018b60e0}, 0x2f1baf3)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc001247a98 sp=0xc001247798 pc=0x3ded372
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x92b8658, 0xc0018b60e0}, 0xc0016f51a0, {0xc000c1d728, 0x18}, {0xc00181f620, 0x1d})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:240 +0x365 fp=0xc001247c20 sp=0xc001247a98 pc=0x6c2ef65
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x92b8658?, 0xc0018b60e0?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x72 fp=0xc001247cc8 sp=0xc001247c20 pc=0x6c2e992
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x92b8658?, 0xc0018b60e0?}, 0xc0018b60f0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc001247d20 sp=0xc001247cc8 pc=0x369fa82
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x92b8658, 0xc0018b60e0}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc001247e68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc001247d98 sp=0xc001247d20 pc=0x369db2e
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x92b85e8, 0xc000b615e0}, 0xc0016f51a0, {0xc000c1d728, 0x18}, {0xc00181f620, 0x1d}, 0x274a48a7800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:196 +0x285 fp=0xc001247ef0 sp=0xc001247d98 pc=0x6c2e705
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:522 +0x7e fp=0xc001247f78 sp=0xc001247ef0 pc=0x6c325fe
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc001247fe0 sp=0xc001247f78 pc=0x4a4be30
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001247fe8 sp=0xc001247fe0 pc=0x2f26cc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 248
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x2f28ae3
rdx    0x0
rdi    0xc93a4a0
rsi    0x80
rbp    0xc0000b5728
rsp    0xc0000b56e0
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xc000582540
r15    0x7
rip    0x2f28ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
untime/select.go:351 +0x8b7 fp=0xc0000c8eb8 sp=0xc0000c8d78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc0013e25a0)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc0000c8fc8 sp=0xc0000c8eb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc0000c8fe0 sp=0xc0000c8fc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000c8fe8 sp=0xc0000c8fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 109
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 193 gp=0xc000e65a40 m=nil [select]:
runtime.gopark(0xc001384f48?, 0x2?, 0xc0?, 0x4d?, 0xc001384ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001384d78 sp=0xc001384d58 pc=0x2f1e50e
runtime.selectgo(0xc001384f48, 0xc001384ef0, 0xc00073d980?, 0x0, 0xc000c0ab70?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc001384eb8 sp=0xc001384d78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc001428240)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc001384fc8 sp=0xc001384eb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc001384fe0 sp=0xc001384fc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001384fe8 sp=0xc001384fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 194
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 233 gp=0xc000ac68c0 m=nil [IO wait]:
runtime.gopark(0xc0010948e0?, 0xc0018f1788?, 0xc5?, 0xbe?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0018f1768 sp=0xc0018f1748 pc=0x2f1e50e
runtime.netpollblock(0x2f88ad8?, 0x2eaa066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc0018f17a0 sp=0xc0018f1768 pc=0x2eddfb7
internal/poll.runtime_pollWait(0x7f8602112c00, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc0018f17c0 sp=0xc0018f17a0 pc=0x2f1cda5
internal/poll.(*pollDesc).wait(0xc0007dd100?, 0xc001341400?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0018f17e8 sp=0xc0018f17c0 pc=0x2fab907
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0007dd100, {0xc001341400, 0x2500, 0x2500})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc0018f1880 sp=0xc0018f17e8 pc=0x2facd39
net.(*netFD).Read(0xc0007dd100, {0xc001341400?, 0xc001341400?, 0x5?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc0018f18c8 sp=0xc0018f1880 pc=0x31712e5
net.(*conn).Read(0xc00100ad18, {0xc001341400?, 0x7f8600572340?, 0x7f8649d3b108?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc0018f1910 sp=0xc0018f18c8 pc=0x3183385
crypto/tls.(*atLeastReader).Read(0xc00113f1d0, {0xc001341400?, 0xa?, 0x859?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc0018f1958 sp=0xc0018f1910 pc=0x31eef1b
bytes.(*Buffer).ReadFrom(0xc000db5ea8, {0x9220320, 0xc00113f1d0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc0018f19b0 sp=0xc0018f1958 pc=0x2fef578
crypto/tls.(*Conn).readFromUntil(0xc000db5c08, {0x9221e00, 0xc00100ad18}, 0xc0018f1a18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0018f19e8 sp=0xc0018f19b0 pc=0x31ef0fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000db5c08, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc0018f1c60 sp=0xc0018f19e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000db5c08, {0xc0013ce000, 0x1000, 0xc000ec2fc0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc0018f1cc8 sp=0xc0018f1c60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc0013258c0, {0xc0013ce000?, 0x32dcb05?, 0x921acc0?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc0018f1d28 sp=0xc0018f1cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc000e1a420)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc0018f1d60 sp=0xc0018f1d28 pc=0x3084943
bufio.(*Reader).Peek(0xc000e1a420, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc0018f1d80 sp=0xc0018f1d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0013258c0)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc0018f1fc8 sp=0xc0018f1d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc0018f1fe0 sp=0xc0018f1fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0018f1fe8 sp=0xc0018f1fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 227
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

rax    0xca
rbx    0x0
rcx    0x2f28ae3
rdx    0x0
rdi    0xc90cc70
rsi    0x80
rbp    0x7ffddd31af40
rsp    0x7ffddd31aef8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xc90a6e0
r15    0x1
rip    0x2f28ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
untime/select.go:351 +0x8b7 fp=0xc0000c8eb8 sp=0xc0000c8d78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc0013e25a0)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc0000c8fc8 sp=0xc0000c8eb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc0000c8fe0 sp=0xc0000c8fc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000c8fe8 sp=0xc0000c8fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 109
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 193 gp=0xc000e65a40 m=nil [select]:
runtime.gopark(0xc001384f48?, 0x2?, 0xc0?, 0x4d?, 0xc001384ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001384d78 sp=0xc001384d58 pc=0x2f1e50e
runtime.selectgo(0xc001384f48, 0xc001384ef0, 0xc00073d980?, 0x0, 0xc000c0ab70?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc001384eb8 sp=0xc001384d78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc001428240)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc001384fc8 sp=0xc001384eb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc001384fe0 sp=0xc001384fc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001384fe8 sp=0xc001384fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 194
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 233 gp=0xc000ac68c0 m=nil [IO wait]:
runtime.gopark(0xc0010948e0?, 0xc0018f1788?, 0xc5?, 0xbe?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0018f1768 sp=0xc0018f1748 pc=0x2f1e50e
runtime.netpollblock(0x2f88ad8?, 0x2eaa066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc0018f17a0 sp=0xc0018f1768 pc=0x2eddfb7
internal/poll.runtime_pollWait(0x7f8602112c00, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc0018f17c0 sp=0xc0018f17a0 pc=0x2f1cda5
internal/poll.(*pollDesc).wait(0xc0007dd100?, 0xc001341400?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0018f17e8 sp=0xc0018f17c0 pc=0x2fab907
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0007dd100, {0xc001341400, 0x2500, 0x2500})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc0018f1880 sp=0xc0018f17e8 pc=0x2facd39
net.(*netFD).Read(0xc0007dd100, {0xc001341400?, 0xc001341400?, 0x5?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc0018f18c8 sp=0xc0018f1880 pc=0x31712e5
net.(*conn).Read(0xc00100ad18, {0xc001341400?, 0x7f8600572340?, 0x7f8649d3b108?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc0018f1910 sp=0xc0018f18c8 pc=0x3183385
crypto/tls.(*atLeastReader).Read(0xc00113f1d0, {0xc001341400?, 0xa?, 0x859?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc0018f1958 sp=0xc0018f1910 pc=0x31eef1b
bytes.(*Buffer).ReadFrom(0xc000db5ea8, {0x9220320, 0xc00113f1d0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc0018f19b0 sp=0xc0018f1958 pc=0x2fef578
crypto/tls.(*Conn).readFromUntil(0xc000db5c08, {0x9221e00, 0xc00100ad18}, 0xc0018f1a18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0018f19e8 sp=0xc0018f19b0 pc=0x31ef0fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000db5c08, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc0018f1c60 sp=0xc0018f19e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000db5c08, {0xc0013ce000, 0x1000, 0xc000ec2fc0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc0018f1cc8 sp=0xc0018f1c60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc0013258c0, {0xc0013ce000?, 0x32dcb05?, 0x921acc0?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc0018f1d28 sp=0xc0018f1cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc000e1a420)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc0018f1d60 sp=0xc0018f1d28 pc=0x3084943
bufio.(*Reader).Peek(0xc000e1a420, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc0018f1d80 sp=0xc0018f1d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0013258c0)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc0018f1fc8 sp=0xc0018f1d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc0018f1fe0 sp=0xc0018f1fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0018f1fe8 sp=0xc0018f1fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 227
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

rax    0xca
rbx    0x0
rcx    0x2f28ae3
rdx    0x0
rdi    0xc90cc70
rsi    0x80
rbp    0x7ffddd31af40
rsp    0x7ffddd31aef8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x1
r14    0xc90a6e0
r15    0x1
rip    0x2f28ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
0f92958 sp=0xc000f92910 pc=0x31eef1b
bytes.(*Buffer).ReadFrom(0xc000fb6628, {0x9220320, 0xc0010c5ef0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000f929b0 sp=0xc000f92958 pc=0x2fef578
crypto/tls.(*Conn).readFromUntil(0xc000fb6388, {0x9221e00, 0xc001114078}, 0xc000f92a18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc000f929e8 sp=0xc000f929b0 pc=0x31ef0fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000fb6388, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000f92c60 sp=0xc000f929e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000fb6388, {0xc0018a4000, 0x1000, 0xc00110be90?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000f92cc8 sp=0xc000f92c60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc0017dfd40, {0xc0018a4000?, 0x32dcb05?, 0x921acc0?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc000f92d28 sp=0xc000f92cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc001886c60)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc000f92d60 sp=0xc000f92d28 pc=0x3084943
bufio.(*Reader).Peek(0xc001886c60, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc000f92d80 sp=0xc000f92d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0017dfd40)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc000f92fc8 sp=0xc000f92d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc000f92fe0 sp=0xc000f92fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000f92fe8 sp=0xc000f92fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 181
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 187 gp=0xc000bbee00 m=nil [select]:
runtime.gopark(0xc0012a1768?, 0x2?, 0x20?, 0x18?, 0xc0012a175c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0012a15e8 sp=0xc0012a15c8 pc=0x2f1e50e
runtime.selectgo(0xc0012a1768, 0xc0012a1758, 0xc000be0080?, 0x0, 0x36ecd5c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc0012a1728 sp=0xc0012a15e8 pc=0x2ef92f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x92b8658, 0xc00004cf50}, 0x8704ddd?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc0012a1798 sp=0xc0012a1728 pc=0x36ec4b6
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x930a440, {0x92b8658, 0xc00004cf50}, 0x2f1baf3)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc0012a1a98 sp=0xc0012a1798 pc=0x3ded372
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x92b8658, 0xc00004cf50}, 0xc0014f41c8, {0xc0001732f0, 0x18}, {0xc001874940, 0xe})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:240 +0x365 fp=0xc0012a1c20 sp=0xc0012a1a98 pc=0x6c2ef65
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x92b8658?, 0xc00004cf50?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x72 fp=0xc0012a1cc8 sp=0xc0012a1c20 pc=0x6c2e992
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x92b8658?, 0xc00004cf50?}, 0xc00004cf60?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc0012a1d20 sp=0xc0012a1cc8 pc=0x369fa82
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x92b8658, 0xc00004cf50}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc0012a1e68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc0012a1d98 sp=0xc0012a1d20 pc=0x369db2e
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x92b85e8, 0xc0008411d0}, 0xc0014f41c8, {0xc0001732f0, 0x18}, {0xc001874940, 0xe}, 0x274a48a7800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:196 +0x285 fp=0xc0012a1ef0 sp=0xc0012a1d98 pc=0x6c2e705
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:522 +0x7e fp=0xc0012a1f78 sp=0xc0012a1ef0 pc=0x6c325fe
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc0012a1fe0 sp=0xc0012a1f78 pc=0x4a4be30
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0012a1fe8 sp=0xc0012a1fe0 pc=0x2f26cc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 176
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x2f28ae3
rdx    0x0
rdi    0xc90cc70
rsi    0x80
rbp    0x7ffe53ff0f80
rsp    0x7ffe53ff0f38
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc000096008
r13    0x1
r14    0xc90a6e0
r15    0x3
rip    0x2f28ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
0f92958 sp=0xc000f92910 pc=0x31eef1b
bytes.(*Buffer).ReadFrom(0xc000fb6628, {0x9220320, 0xc0010c5ef0})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc000f929b0 sp=0xc000f92958 pc=0x2fef578
crypto/tls.(*Conn).readFromUntil(0xc000fb6388, {0x9221e00, 0xc001114078}, 0xc000f92a18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc000f929e8 sp=0xc000f929b0 pc=0x31ef0fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000fb6388, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc000f92c60 sp=0xc000f929e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000fb6388, {0xc0018a4000, 0x1000, 0xc00110be90?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc000f92cc8 sp=0xc000f92c60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc0017dfd40, {0xc0018a4000?, 0x32dcb05?, 0x921acc0?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc000f92d28 sp=0xc000f92cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc001886c60)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc000f92d60 sp=0xc000f92d28 pc=0x3084943
bufio.(*Reader).Peek(0xc001886c60, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc000f92d80 sp=0xc000f92d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0017dfd40)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc000f92fc8 sp=0xc000f92d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc000f92fe0 sp=0xc000f92fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000f92fe8 sp=0xc000f92fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 181
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 187 gp=0xc000bbee00 m=nil [select]:
runtime.gopark(0xc0012a1768?, 0x2?, 0x20?, 0x18?, 0xc0012a175c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0012a15e8 sp=0xc0012a15c8 pc=0x2f1e50e
runtime.selectgo(0xc0012a1768, 0xc0012a1758, 0xc000be0080?, 0x0, 0x36ecd5c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc0012a1728 sp=0xc0012a15e8 pc=0x2ef92f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x92b8658, 0xc00004cf50}, 0x8704ddd?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc0012a1798 sp=0xc0012a1728 pc=0x36ec4b6
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x930a440, {0x92b8658, 0xc00004cf50}, 0x2f1baf3)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc0012a1a98 sp=0xc0012a1798 pc=0x3ded372
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x92b8658, 0xc00004cf50}, 0xc0014f41c8, {0xc0001732f0, 0x18}, {0xc001874940, 0xe})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:240 +0x365 fp=0xc0012a1c20 sp=0xc0012a1a98 pc=0x6c2ef65
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x92b8658?, 0xc00004cf50?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x72 fp=0xc0012a1cc8 sp=0xc0012a1c20 pc=0x6c2e992
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x92b8658?, 0xc00004cf50?}, 0xc00004cf60?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc0012a1d20 sp=0xc0012a1cc8 pc=0x369fa82
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x92b8658, 0xc00004cf50}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc0012a1e68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc0012a1d98 sp=0xc0012a1d20 pc=0x369db2e
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x92b85e8, 0xc0008411d0}, 0xc0014f41c8, {0xc0001732f0, 0x18}, {0xc001874940, 0xe}, 0x274a48a7800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:196 +0x285 fp=0xc0012a1ef0 sp=0xc0012a1d98 pc=0x6c2e705
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:522 +0x7e fp=0xc0012a1f78 sp=0xc0012a1ef0 pc=0x6c325fe
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc0012a1fe0 sp=0xc0012a1f78 pc=0x4a4be30
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0012a1fe8 sp=0xc0012a1fe0 pc=0x2f26cc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 176
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x2f28ae3
rdx    0x0
rdi    0xc90cc70
rsi    0x80
rbp    0x7ffe53ff0f80
rsp    0x7ffe53ff0f38
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0xc000096008
r13    0x1
r14    0xc90a6e0
r15    0x3
rip    0x2f28ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
Contributing tests (4)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group3
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2068591174670618624
n/a2068520879767162880
n/a2068465010476060672
n/a2068402345783332864
n/a2068004614032068608
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
full failure pattern:
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e811.76%8 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 10 · Jun 20: 7 · Jun 21: 6none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-blwhnk"
Unexpected error:
    <*fmt.wrapError | 0xc000886a60>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-4j25v.apps.aro.f2b3g0f2g4h5f7w.mhke.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-4j25v.apps.aro.f2b3g0f2g4h5f7w.mhke.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-blwhnk"
Unexpected error:
    <*fmt.wrapError | 0xc000886a60>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-4j25v.apps.aro.f2b3g0f2g4h5f7w.mhke.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-4j25v.apps.aro.f2b3g0f2g4h5f7w.mhke.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-sdplf7"
Unexpected error:
    <*fmt.wrapError | 0xc000706300>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-zxdd2.apps.aro.i9r9j8o2x4u9b9e.7lui.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-zxdd2.apps.aro.i9r9j8o2x4u9b9e.7lui.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-sdplf7"
Unexpected error:
    <*fmt.wrapError | 0xc000706300>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-zxdd2.apps.aro.i9r9j8o2x4u9b9e.7lui.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-zxdd2.apps.aro.i9r9j8o2x4u9b9e.7lui.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-pvzn6h"
Unexpected error:
    <*fmt.wrapError | 0xc000986860>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-zwp5z.apps.aro.p2d4i7x8i2d6g9t.og38.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-zwp5z.apps.aro.p2d4i7x8i2d6g9t.og38.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-pvzn6h"
Unexpected error:
    <*fmt.wrapError | 0xc000986860>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-zwp5z.apps.aro.p2d4i7x8i2d6g9t.og38.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-zwp5z.apps.aro.p2d4i7x8i2d6g9t.og38.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.226
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.205
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.195
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.214
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.03
Affected runs (8)
Date (UTC)Associated PRProw job
n/a2068641277968125952
n/a2068591174670618624
n/a2068520879767162880
n/a2068465010476060672
n/a2068402345783332864
n/a2068082178301890560
n/a2068004614032068608
n/a2067949798622760960
VerifySimpleWebApp failed: route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
full failure pattern:
VerifySimpleWebApp failed: route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e68.82%6 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 2 · Jun 20: 4 · Jun 21: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001dec1b0>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-g7mbq.apps.aro.cilium-cl.oryn.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-g7mbq.apps.aro.cilium-cl.oryn.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001dec1b0>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-g7mbq.apps.aro.cilium-cl.oryn.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-g7mbq.apps.aro.cilium-cl.oryn.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000b48ab0>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-qmtsc.apps.aro.cilium-cl.afaw.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-qmtsc.apps.aro.cilium-cl.afaw.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000b48ab0>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-qmtsc.apps.aro.cilium-cl.afaw.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-qmtsc.apps.aro.cilium-cl.afaw.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0011601b0>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-f9l8r.apps.aro.cilium-cluster.9wlt.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-f9l8r.apps.aro.cilium-cluster.9wlt.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0011601b0>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-f9l8r.apps.aro.cilium-cluster.9wlt.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-f9l8r.apps.aro.cilium-cluster.9wlt.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin6
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully4
Affected runs (6)
Date (UTC)Associated PRProw job
n/a2068641277968125952
n/a2068591174670618624
n/a2068520879767162880
n/a2068465010476060672
n/a2068402345783332864
n/a2068082178301890560
dial tcp: lookup console-openshift-console
full failure pattern:
dial tcp: lookup console-openshift-console
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e57.35%5 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 2 · Jun 21: 2none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc000bdaf00>: 
    dial tcp: lookup console-openshift-console.apps.aro.i9f1o4d9p7n0g1u.dh5j.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc000bdaf00>: 
    dial tcp: lookup console-openshift-console.apps.aro.i9f1o4d9p7n0g1u.dh5j.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc000b7eb40>: 
    dial tcp: lookup console-openshift-console.apps.aro.m1c4h3t4a2g8i8d.4hwy.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc000b7eb40>: 
    dial tcp: lookup console-openshift-console.apps.aro.m1c4h3t4a2g8i8d.4hwy.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.003s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc0008c56d0>: 
    dial tcp: lookup console-openshift-console.apps.aro.f4y0j8z4x0b5v2h.j4r9.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.003s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc0008c56d0>: 
    dial tcp: lookup console-openshift-console.apps.aro.f4y0j8z4x0b5v2h.j4r9.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates5
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2068641277968125952
n/a2068520879767162880
n/a2068465010476060672
n/a2068402345783332864
n/a2068082178301890560
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...
full failure pattern:
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e45.88%4 of 68 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 2 · Jun 18: 0 · Jun 19: 8 · Jun 20: 1 · Jun 21: 1none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000fea4b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-gj9qbc" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000fea4b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-gj9qbc" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000cf4240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-t2dlwt" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000cf4240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-t2dlwt" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012fe150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-6b2hnp" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012fe150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-6b2hnp" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
Contributing tests (11)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2068641277968125952
n/a2068465010476060672
n/a2067929498837323776
n/a2067198813889630208
ERROR CODE: InvalidTemplateDeployment; provider Microsoft.Compute
full failure pattern:
ERROR CODE: InvalidTemplateDeployment; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e45.88%4 of 68 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsJun 14: 2 · Jun 15: 3 · Jun 16: 1 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc000acb640>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-dcwq4p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-dcwq4p/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is '08aa4ae2-50da-40f0-9df7-0ce331ab97cf'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22uksouth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc000acb640>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-dcwq4p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-dcwq4p/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is '08aa4ae2-50da-40f0-9df7-0ce331ab97cf'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22uksouth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc00132d5a0>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-pztdvd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-pztdvd/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is 'f6a5d54e-c1c0-4fbe-b749-d7c681bd8016'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22uksouth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc00132d5a0>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-pztdvd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-pztdvd/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is 'f6a5d54e-c1c0-4fbe-b749-d7c681bd8016'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: uksouth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22uksouth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc000be6720>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-c55hn7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-c55hn7/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is 'c2e9b250-dda1-479b-a710-c7ed5bd06b48'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: SwitzerlandNorth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22SwitzerlandNorth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:130]: failed to deploy test VM
Unexpected error:
    <*fmt.wrapError | 0xc000be6720>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-c55hn7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-c55hn7/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Compute/virtualMachines (2023-03-01)' reported preflight validation errors. Tracking id is 'c2e9b250-dda1-479b-a710-c7ed5bd06b48'. See inner errors for details.",
        "details": [
          {
            "code": "QuotaExceeded",
            "message": "Operation could not be completed as it results in exceeding approved standardDSv5Family Cores quota. Additional details - Deployment Model: Resource Manager, Location: SwitzerlandNorth, Current Limit: 0, Current Usage: 0, Additional Required: 2, (Minimum) New Limit Required: 2. Setup Alerts when Quota reaches threshold. Learn more at https://aka.ms/quotamonitoringalerting . Submit a request for Quota increase at https://aka.ms/ProdportalCRP/#blade/Microsoft_Azure_Capacity/UsageAndQuota.ReactView/Parameters/%7B%22subscriptionId%22:%22XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%22,%22command%22:%22openQuotaApprovalBlade%22,%22quotas%22:[%7B%22location%22:%22SwitzerlandNorth%22,%22providerId%22:%22Microsoft.Compute%22,%22resourceName%22:%22standardDSv5Family%22,%22quotaRequest%22:%7B%22properties%22:%7B%22limit%22:2,%22unit%22:%22Count%22,%22name%22:%7B%22value%22:%22standardDSv5Family%22%7D%7D%7D%7D]%7D by specifying parameters listed in the ‘Details’ section for deployment to succeed. Please read more about quota limits at https://docs.microsoft.com/en-us/azure/azure-supportability/per-vm-quota-requests"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (4)
Date (UTC)Associated PRProw job
n/a2066702365632761856
n/a2066657066361032704
n/a2066611850006499328
n/a2066339954090315776
cluster has not yet reached Succeeded state
full failure pattern:
cluster has not yet reached Succeeded state
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e34.41%3 of 68 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 1 · Jun 18: 0 · Jun 19: 2 · Jun 20: 0 · Jun 21: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.008s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.008s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation3
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2068063035146113024
n/a2067929498837323776
n/a2067198813889630208
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
full failure pattern:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e34.41%3 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 0 · Jun 21: 2none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.000s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:406 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.000s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:406 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.002s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:406 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.002s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:406 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:406 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:406 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: ...
to be empty
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP3
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2068591174670618624
n/a2068520879767162880
n/a2067949798622760960
Cluster should become ready within '20.000000' minutes
full failure pattern:
Cluster should become ready within '20.000000' minutes
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e22.94%2 of 68 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 2 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.005s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.005s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068063035146113024
n/a2067929498837323776
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
full failure pattern:
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e22.94%2 of 68 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 2 · Jun 20: 0 · Jun 21: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:191]: failed to create node pool version 4.19.7 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc001776480>: 
    failed waiting for nodepool="np-ver-4-19-7" for cluster "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-9r69jz" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:191]: failed to create node pool version 4.19.7 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc001776480>: 
    failed waiting for nodepool="np-ver-4-19-7" for cluster "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-9r69jz" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:138]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc001027260>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-cnkjd2" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:138]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc001027260>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-cnkjd2" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068082178301890560
n/a2067949798622760960
ERROR CODE: InternalServerError; detail message Get "<url>": net/http: TLS handshake timeout; provider Microsoft.Compute
full failure pattern:
ERROR CODE: InternalServerError; detail message Get "<url>": net/http: TLS handshake timeout; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e22.94%2 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 0 · Jun 21: 1none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0009535a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-sfdjsl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9494c8db-21a8-45dc-a144-9909ebdbc9f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9494c8db-21a8-45dc-a144-9909ebdbc9f1",
      "name": "9494c8db-21a8-45dc-a144-9909ebdbc9f1",
      "status": "Failed",
      "startTime": "2026-06-21T07:22:15.977086685Z",
      "endTime": "2026-06-21T07:23:35.108555439Z",
      "error": {
        "code": "InternalServerError",
        "message": "Get \"https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus?%24filter=location+eq+%27brazilsouth%27\u0026api-version=2021-07-01\": net/http: TLS handshake timeout"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0009535a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-sfdjsl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9494c8db-21a8-45dc-a144-9909ebdbc9f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/9494c8db-21a8-45dc-a144-9909ebdbc9f1",
      "name": "9494c8db-21a8-45dc-a144-9909ebdbc9f1",
      "status": "Failed",
      "startTime": "2026-06-21T07:22:15.977086685Z",
      "endTime": "2026-06-21T07:23:35.108555439Z",
      "error": {
        "code": "InternalServerError",
        "message": "Get \"https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus?%24filter=location+eq+%27brazilsouth%27\u0026api-version=2021-07-01\": net/http: TLS handshake timeout"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:168]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc001910860>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-td4tgf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "name": "56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "status": "Failed",
      "startTime": "2026-06-19T18:20:56.860797749Z",
      "endTime": "2026-06-19T18:22:33.081221493Z",
      "error": {
        "code": "InternalServerError",
        "message": "Get \"https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus?%24filter=location+eq+%27brazilsouth%27\u0026api-version=2021-07-01\": net/http: TLS handshake timeout"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:168]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc001910860>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-td4tgf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "name": "56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "status": "Failed",
      "startTime": "2026-06-19T18:20:56.860797749Z",
      "endTime": "2026-06-19T18:22:33.081221493Z",
      "error": {
        "code": "InternalServerError",
        "message": "Get \"https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus?%24filter=location+eq+%27brazilsouth%27\u0026api-version=2021-07-01\": net/http: TLS handshake timeout"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068591174670618624
n/a2068004614032068608
add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
full failure pattern:
add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e22.94%2 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 1 · Jun 21: 1none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:132]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc000e89980>: 
    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:132]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc000e89980>: 
    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:132]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc0007bf3b0>: 
    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:132]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc0007bf3b0>: 
    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068520879767162880
n/a2068216024569745408
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Pro...
full failure pattern:
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e22.94%2 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 1 · Jun 21: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc0013bc3f0>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc0013bc3f0>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc00101e000>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc00101e000>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068465010476060672
n/a2068082178301890560
couldn't get current server API group list: Get "<url>": dial tcp <ip>:<port>: i/o timeout
full failure pattern:
couldn't get current server API group list: Get "<url>": dial tcp <ip>:<port>: i/o timeout
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e22.94%2 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 1 · Jun 21: 1none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.008s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:403 with:
kubectl get clusteroperators should succeed from authorized VM
Unexpected error:
    <*errors.errorString | 0xc001703260>: 
    E0621 11:01:15.116862    3256 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.s6u4e9h6n4x5p4f.go5d.brazilsouth.aroapp-hcp.io:443/api?timeout=32s\": dial tcp 74.163.139.192:443: i/o timeout"
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.008s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:403 with:
kubectl get clusteroperators should succeed from authorized VM
Unexpected error:
    <*errors.errorString | 0xc001703260>: 
    E0621 11:01:15.116862    3256 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.s6u4e9h6n4x5p4f.go5d.brazilsouth.aroapp-hcp.io:443/api?timeout=32s\": dial tcp 74.163.139.192:443: i/o timeout"
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 605.076s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:403 with:
kubectl get clusteroperators should succeed from authorized VM
Unexpected error:
    <*errors.errorString | 0xc000a7a390>: 
    E0621 00:06:17.087501    3391 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.h5h5i6l8r7c3p7a.u2v6.brazilsouth.aroapp-hcp.io:443/api?timeout=32s\": dial tcp 74.163.139.192:443: i/o timeout"
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 605.076s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:403 with:
kubectl get clusteroperators should succeed from authorized VM
Unexpected error:
    <*errors.errorString | 0xc000a7a390>: 
    E0621 00:06:17.087501    3391 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.h5h5i6l8r7c3p7a.u2v6.brazilsouth.aroapp-hcp.io:443/api?timeout=32s\": dial tcp 74.163.139.192:443: i/o timeout"
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068641277968125952
n/a2068465010476060672
failed to wait for first cluster "<cluster>" to complete creation (timeout '20.000000' minutes)
full failure pattern:
failed to wait for first cluster "<cluster>" to complete creation (timeout '20.000000' minutes)
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e22.94%2 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 2 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068063035146113024
n/a2067929498837323776
x509: certificate signed by unknown authority
full failure pattern:
x509: certificate signed by unknown authority
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e22.94%2 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 1 · Jun 18: 0 · Jun 19: 1 · Jun 20: 0 · Jun 21: 0none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.003s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.003s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.002s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2067820806943019008
n/a2067363950726483968
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
full failure pattern:
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 3 · Jun 18: 0 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:169]: failed to get admin REST config for patch-tags cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000f70960>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-dwtz8f" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-dwtz8f, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:169]: failed to get admin REST config for patch-tags cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000f70960>: 
    failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-dwtz8f" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster patch-tags-cluster in resource group patch-tags-dwtz8f, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:93]: failed to get admin REST config for cluster "cilium-cl"
Unexpected error:
    <*fmt.wrapErrors | 0xc000040e40>: 
    failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-tbdjlb" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cl in resource group cni-cilium-tbdjlb, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:93]: failed to get admin REST config for cluster "cilium-cl"
Unexpected error:
    <*fmt.wrapErrors | 0xc000040e40>: 
    failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-tbdjlb" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cilium-cl in resource group cni-cilium-tbdjlb, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:99]: failed to get admin REST config for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc001394750>: 
    failed waiting for hcpCluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-cqbsn8" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-cqbsn8, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:99]: failed to get admin REST config for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc001394750>: 
    failed waiting for hcpCluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-cqbsn8" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-cqbsn8, error: context deadline exceeded
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067198813889630208
tls: failed to verify certificate: x509: certificate signed by unknown authority
full failure pattern:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-4rkx7j"
Unexpected error:
    <*fmt.wrapError | 0xc000e0a620>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-9rwln.apps.aro.s0y5l7a1j6m3k3z.r0ib.australiaeast.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-4rkx7j"
Unexpected error:
    <*fmt.wrapError | 0xc000e0a620>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-9rwln.apps.aro.s0y5l7a1j6m3k3z.r0ib.australiaeast.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068063774274752512
node pool <nodepool> verification failed: <node> (release images unchanged: [quay
full failure pattern:
node pool <nodepool> verification failed: <node> (release images unchanged: [quay
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:220]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.errorString | 0xc00081af90>: 
    node pool upgrade verification failed: e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk (release images unchanged: [quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258 quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f quay.io/openshift/origin-insights-runtime-extractor:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13 quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184 quay.io/openshift/origin-kube-rbac-proxy:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9 arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1 quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef quay.io/openshift/origin-insights-runtime-exporter:latest]); e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq (not ready); nodes=[{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk","ready":true,"containerRuntimeVersion":"cri-o://1.34.2-2.rhaos4.21.gitc8e8b46.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-4xsj8","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44c807272cb410a84d347dda34dd77990697384e450b7b832e533e2a2352de76","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48c6e2725865ef2b8b7d6dc501c1778d681ed1eaf8bccad0d978c163b36a8ad9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad2768f4eed20ecddef9652e72c2536a9c5a129a733ccb8c423b8421b1d047bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd7c7b127ef0c7067d52f4ae8f3f95f878ec1c8b78a64fbb89f184824211ed80","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1110f6e8d66ba68224f9a7eedf1baa54de2d0c58332f611862242fe8ff311295","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1459af715cdfb7e21e056d793478f4387d077ad3d4d4e095c250c10eef809631","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29dd11a9e6e7344676611823c362862070b2ebc080793514aa14178527bfa7f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f35aba18e4a81d2e301162430d451d147378a5bd74a711135bead02921256554","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e18b7e1a7e67c4cff5e27d62c7de71968108b4366f8e30619fe6d277288dd63","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b61249fb7f26b253b8e634ef8cc47aa599b108e74651dba2c0dbda94df82f30","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab245b1c592c3179b63e3a79fa641f8125c16a29a67338f9adb489fb85af87b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cd91d2f0e56a7e3484ad8718845f9747c94180b0ee975fb62ffc75df214577b","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16139e0510e8b95cab7c2645e63f08d069c15ed64553e5133b99e59736ead4df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84113d4695be785283aacfc42fb9905bb7cd9233c4157f7d09726e641ea0d648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:761ccd3ced18e3e3d3760ece8e13bc07eb6119fdc05e4e5f7677d2695ab99ff4","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d6207a89a90d9f3408da1ec88e8c13c8c23330faf4c82be14c1e07b9ddb44a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-9txmg","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00f6ba985494302f2d431ad27b9bc6300d0c0d2dd0433e6403525e427a3341fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5469f5ec20ac2e7a2d82e84767c1c635671fce4ae3d7c27402289254f1183e04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a127e2a436b84937d2d970f3c316d82f61da498f04ec6036a91ad4284403f913","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d69fdb4df0385545fea609e6665de6c4b6dd4763b73a35a9d49456f879987925","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84493322067d59517be692eb5b1ffc90ae87f1a382459de635a69db4520e2750","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0476037c9c3b76a0135bc565733743cda8777479dc92a5cf2e6b79e999d35e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq","ready":false,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:220]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.errorString | 0xc00081af90>: 
    node pool upgrade verification failed: e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk (release images unchanged: [quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258 quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f quay.io/openshift/origin-insights-runtime-extractor:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13 quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184 quay.io/openshift/origin-kube-rbac-proxy:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9 arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1 quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef quay.io/openshift/origin-insights-runtime-exporter:latest]); e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq (not ready); nodes=[{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk","ready":true,"containerRuntimeVersion":"cri-o://1.34.2-2.rhaos4.21.gitc8e8b46.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-4xsj8","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44c807272cb410a84d347dda34dd77990697384e450b7b832e533e2a2352de76","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48c6e2725865ef2b8b7d6dc501c1778d681ed1eaf8bccad0d978c163b36a8ad9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad2768f4eed20ecddef9652e72c2536a9c5a129a733ccb8c423b8421b1d047bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd7c7b127ef0c7067d52f4ae8f3f95f878ec1c8b78a64fbb89f184824211ed80","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1110f6e8d66ba68224f9a7eedf1baa54de2d0c58332f611862242fe8ff311295","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1459af715cdfb7e21e056d793478f4387d077ad3d4d4e095c250c10eef809631","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29dd11a9e6e7344676611823c362862070b2ebc080793514aa14178527bfa7f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f35aba18e4a81d2e301162430d451d147378a5bd74a711135bead02921256554","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e18b7e1a7e67c4cff5e27d62c7de71968108b4366f8e30619fe6d277288dd63","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b61249fb7f26b253b8e634ef8cc47aa599b108e74651dba2c0dbda94df82f30","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab245b1c592c3179b63e3a79fa641f8125c16a29a67338f9adb489fb85af87b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cd91d2f0e56a7e3484ad8718845f9747c94180b0ee975fb62ffc75df214577b","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16139e0510e8b95cab7c2645e63f08d069c15ed64553e5133b99e59736ead4df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84113d4695be785283aacfc42fb9905bb7cd9233c4157f7d09726e641ea0d648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:761ccd3ced18e3e3d3760ece8e13bc07eb6119fdc05e4e5f7677d2695ab99ff4","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d6207a89a90d9f3408da1ec88e8c13c8c23330faf4c82be14c1e07b9ddb44a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-9txmg","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00f6ba985494302f2d431ad27b9bc6300d0c0d2dd0433e6403525e427a3341fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5469f5ec20ac2e7a2d82e84767c1c635671fce4ae3d7c27402289254f1183e04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a127e2a436b84937d2d970f3c316d82f61da498f04ec6036a91ad4284403f913","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d69fdb4df0385545fea609e6665de6c4b6dd4763b73a35a9d49456f879987925","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84493322067d59517be692eb5b1ffc90ae87f1a382459de635a69db4520e2750","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0476037c9c3b76a0135bc565733743cda8777479dc92a5cf2e6b79e999d35e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq","ready":false,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678"]}]
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067820806943019008
ERROR CODE: DeploymentNotFound
full failure pattern:
ERROR CODE: DeploymentNotFound
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 1 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-20-fw7f2d"
Unexpected error:
    <*fmt.wrapError | 0xc000f9b3c0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-fw7f2d-k89rj5" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-fw7f2d-k89rj5/operationStatuses/08584198113386810784
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-cluster-zstream-4-20-fw7f2d-k89rj5' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:92]: failed to create customer resources for z-stream cluster "cluster-zstream-4-20-fw7f2d"
Unexpected error:
    <*fmt.wrapError | 0xc000f9b3c0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-20-fw7f2d-k89rj5" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-20-fw7f2d-k89rj5/operationStatuses/08584198113386810784
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-cluster-zstream-4-20-fw7f2d-k89rj5' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067598465302532096
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v4.20.0-0.nightly-multi-2026-06-21-183852-nightly' doesn't exist; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v4.20.0-0.nightly-multi-2026-06-21-183852-nightly' doesn't exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 0 · Jun 21: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f86220>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-s2fhzz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-s2fhzz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.20.0-0.nightly-multi-2026-06-21-183852-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f86220>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-s2fhzz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-s2fhzz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.20.0-0.nightly-multi-2026-06-21-183852-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to use workload identity via the cluster OIDC issuer URL1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068831445857603584
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v4.23.0-0.nightly-multi-2026-06-15-224233-nightly' doesn't exist; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Version 'openshift-v4.23.0-0.nightly-multi-2026-06-15-224233-nightly' doesn't exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 1 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-jhhdtf/cluster-nightly-4-23-qw58nx should provision
Unexpected error:
    <*fmt.wrapError | 0xc0006afaa0>: 
    failed starting cluster creation "cluster-nightly-4-23-qw58nx" in resourcegroup="rg-nightly-4-23-jhhdtf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-jhhdtf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-qw58nx
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.23.0-0.nightly-multi-2026-06-15-224233-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: Swift cluster rg-nightly-4-23-jhhdtf/cluster-nightly-4-23-qw58nx should provision
Unexpected error:
    <*fmt.wrapError | 0xc0006afaa0>: 
    failed starting cluster creation "cluster-nightly-4-23-qw58nx" in resourcegroup="rg-nightly-4-23-jhhdtf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-jhhdtf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-qw58nx
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Version 'openshift-v4.23.0-0.nightly-multi-2026-06-15-224233-nightly' doesn't exist"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.231
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2066657066361032704
Get "<url>": dial tcp <ip>:<port>: i/o timeout
full failure pattern:
Get "<url>": dial tcp <ip>:<port>: i/o timeout
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to get ClusterVersion resource
Unexpected error:
    <*url.Error | 0xc000b63410>: 
    Get "https://api.i2o1x2r3h8b1j4g.mbnd.brazilsouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to get ClusterVersion resource
Unexpected error:
    <*url.Error | 0xc000b63410>: 
    Get "https://api.i2o1x2r3h8b1j4g.mbnd.brazilsouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067949798622760960
Get "<url>": net/http: TLS handshake timeout
full failure pattern:
Get "<url>": net/http: TLS handshake timeout
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:162]: failed to list nodes after AZ nodepool creation
Unexpected error:
    <*url.Error | 0xc0016e2240>: 
    Get "https://api.f2e6d6p6j1q8o4d.91nt.uksouth.aroapp-hcp.io:443/api/v1/nodes": net/http: TLS handshake timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:162]: failed to list nodes after AZ nodepool creation
Unexpected error:
    <*url.Error | 0xc0016e2240>: 
    Get "https://api.f2e6d6p6j1q8o4d.91nt.uksouth.aroapp-hcp.io:443/api/v1/nodes": net/http: TLS handshake timeout
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068106661532274688
Post "<url>": dial tcp <ip>:<port>: i/o timeout
full failure pattern:
Post "<url>": dial tcp <ip>:<port>: i/o timeout
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:236]: should be able to create SelfSubjectReview for admin credential 1
Unexpected error:
    <*url.Error | 0xc00130c030>: 
    Post "https://api.u6k7t4d6d9b4a5c.vf44.brazilsouth.aroapp-hcp.io:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:236]: should be able to create SelfSubjectReview for admin credential 1
Unexpected error:
    <*url.Error | 0xc00130c030>: 
    Post "https://api.u6k7t4d6d9b4a5c.vf44.brazilsouth.aroapp-hcp.io:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068082178301890560
Should fail with EOF error indicating blocked connection
full failure pattern:
Should fail with EOF error indicating blocked connection
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 1 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:190]: Should fail with EOF error indicating blocked connection
Expected
    <string>: ...
to contain substring
    <string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:190]: Should fail with EOF error indicating blocked connection
Expected
    <string>: ...
to contain substring
    <string>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067744257157894144
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 1 · Jun 18: 0 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:225]: failed to update node pool np-1 with new label and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc00134f830>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-n48kcf" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-n48kcf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:225]: failed to update node pool np-1 with new label and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc00134f830>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-n48kcf" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-n48kcf, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067198813889630208
failed to create namespace: Post "<url>": dial tcp <ip>:<port>: i/o timeout
full failure pattern:
failed to create namespace: Post "<url>": dial tcp <ip>:<port>: i/o timeout
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 1 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-qqg5xh"
Unexpected error:
    <*fmt.wrapError | 0xc000b1a060>: 
    failed to create namespace: Post "https://api.q8q3f7q1h7j5r8s.w6pe.brazilsouth.aroapp-hcp.io:443/api/v1/namespaces": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-qqg5xh"
Unexpected error:
    <*fmt.wrapError | 0xc000b1a060>: 
    failed to create namespace: Post "https://api.q8q3f7q1h7j5r8s.w6pe.brazilsouth.aroapp-hcp.io:443/api/v1/namespaces": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068402345783332864
failed to poll admin credential 1 to completion
full failure pattern:
failed to poll admin credential 1 to completion
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 1 · Jun 18: 0 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:191]: failed to poll admin credential 1 to completion
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:191]: failed to poll admin credential 1 to completion
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067198813889630208
node pool <nodepool> verification failed: <node> (version 4
full failure pattern:
node pool <nodepool> verification failed: <node> (version 4
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.47%1 of 68 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 1 · Jun 18: 0 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:517]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.errorString | 0xc00149cc90>: 
    node pool upgrade verification failed: q4n4i3p4b4e6x0b-nps-420-422-gxph5-9xn25 (version 4.20 not in same minor as expected 4.22.1); nodes=[{"name":"q4n4i3p4b4e6x0b-nps-420-422-gxph5-9xn25","ready":true,"containerRuntimeVersion":"cri-o://1.33.12-3.rhaos4.20.git64c6a00.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05d60b3807f2273ac84ebfceee6f8342bf2eb7d14f87d4ba346e3911a6a48d62","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98457921e71a329fbdefb60e1bbd3de80b145489221fbe9150eb90e1abed9786","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13c7f68070664781a1a338f565f1f24425a764f28557280a0bbf9da51a5172a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b519469b498b28347d0a504ef69d2f480cc4431e50d3972933a86147d215d1e3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:65f250c7337f5765db69690b4530e31a6bf4050d9140c4c85527e7c26e77b7cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:836532b41550b915235e842a4c2215953e78dac14357e6566cfdaa72eafe5dfd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40234292b7d4c28149b5493acbaeb608ac59326dbeb53d3cfdcdc90f2c98928b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad892d1fda138cbefcd7b1bb79c47897fb31c0a09429d53001aa777d9af7b8d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:918ffd5c62760ee30d3a6874c5b98cf8a82e2c471fb5a9df501909c4d671e9fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa2835a5b193048ba6b300577088213cc90621c32cf6d8ebfe3d7e2fcee12dbc","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ed76865cd6513d71c46844d3027f3f7b161835b360d2e69b59e297019ecd3fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9e0bd9b962e5326507ec7fd7df5e397e1e88c9f7117bdace555e9eb6cf90447","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0edb65f15d6600020bf0056db95a453688976a6a259251afabd0bd04c18c678a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8806ad37ac4897064437a6a466579fbb65e50d3c06253cb1a56e125455b01cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:236da75c5998805911c6789b80ac704875a79b0abd57a11485218d399ae24be6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d4102507f29eb70d722c8478eb243a4b1913a74f937ed4c1c91f1df5d124ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:801b5a471e7d6c506eea3f249c0f771e2becd4c1a33c03f8355f249a4f7d5f6b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f454167c3e9bea08e362726c4ca4f9f11c8c1c9e8ccfb0612496689f109c1b5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:810098f0663c4a402b3a335cc995a91dac1c4a822b2762f874ba2591fc73f76e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf233f0f2e47df07d74e53d391de580e8db76a58bbda92919feda304a0360c2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8441db4cbe6043f6a4ec0206d37cce75679dbd655310345033d1bce3975760ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cbd95a1f4bc47193aaba2ee9866fc5fd8eab5a1e6fdce8e4b343aa8c49ed6ae5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9c967279d60eeeba741ce8274a719ec4f669aad4eca6eca260cd650eeca9a14","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef9a9ac340cda9a1ea2e571dd741b9df56989d937bf020071efbde3d0333a9f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f8b08e39afa2f499c26c7b286274c4a6430db49fbb87e48eaea935c96de23ff","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98b5f93d5ab68a886c325add352d6675a1fd037db44763c270026e9bd66c28f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cb360c71a907253defcb8fcef0cc29002f23bf6b60926027a6a686749ab809f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c3db74d37a5044a0bbb0c1ef15c7124524891d7af741a98c4477c6d8a5858cd0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5143315a1b4121c00a8240b51db968df01d8c33e49497a3bb94879d218bf64d9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df2cf7a02f8cb20d384ebd144a90c1e45a49b8e33cc81cf9c06dc31f926f0ddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abc8265bad76d67cbb8b6c5c3b674ca1d80180611b2bc053024237aa85367e1b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be44c06ff946fafc0a137cb020af2509b3f2d1f7d8c8b081697ec3483df622d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3ee649c0980af383a375ff9b8287f6f81617ab04eab7b21cea2310ec880070c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dac62b77d6871d5263cf3fc1847466343ce68d80bea6abfb4ba9d70a26b980f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b2fe398a7fe679582f8a653eb139ab8f7b1658ce79daa7d25c8d4aded5dfb9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac6aeac333c22ad2b6c42a40a87067e0acd93d9c45cb4af925c52b87229a7b67","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71d5cf9c12963df3b4a478bc77110df9795a7970d9e74174d4cbf540d89a2b37","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a80616a75bec0e677d4594bf5a7122c792abdacf6e87cd166bb6d2ea616d4af2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1e827b8ef69b1f0a71733c3e646344ccb5fb7e5af146016fd2197ef1c3b560d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ae8602190ff0312684494fb8c5f2aa28c1621837a91e2e765f6e85229cc8c118","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ce5a7e16f173cf9fa5a6810bb784324074d758ccf57f26d47ace826346b7a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5ac3a253cc1e80575f9340f5221edb6b0084f9cf26efea6f582c6e6efd674b4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0dbb2aa398c609b7d717e3df90d57ca64d83211f8303481c3df0fb5f7f609901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9e2af850e2160b07ec78fab74be95928ea1848abbf13aa6f9fe6bc66d8272ab1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:117a9ef195508484c302bb9e03e4e9330f97029ea97eabbe6d00a48fd6640aed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b62893ea3da92e93050dd06e3a47ccb74e18178999b4217f2a4b812b7f7e5126","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19aed0b6d293e76c12cab014c1f37860745beed92b005975cdc0453dc31df3c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82ee0e1d7f7383a28694d18751fa691f1fcfca60a690e081c7b9a30cc4a3b541","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0068453dc71c624460a368872a30785d165d184ad2fcc70159540c35cf564a04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:34853aec39e7fa4e06d003e2f4ad908ca682bca79428755b6c2bd8c4ab3aa5f9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f5c22305bba756410991c00ab3142bb5f24b5aac3ac992afc0d0cf8f66698d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b276ebfe0115876e7e2a1ce1ec99e9762da451104b6d55db5ebc341da8d5f266","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8bcdf087244f707380628f86b8c0c35eeefe870f5107f4693b83632508fbb9d2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96a94e57ccff7cac3eba0a3c64ddca0371624f31b025bccab550633969249361","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a5a30de78b92f1f0f41f1a2188bb3e94318139c9b6e5fab900a002fa766c67b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82f171298a467819ed387a4b1861d3c86d06e417175592bc6e018c756f1f884c","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ae36b4ba7f205b81ef595f0c59c7a1548520d29309598a37328f40708449f2e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd4f32783510721d8e1a61884428d56f44b175c8786498f13f6d554f969f1e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ada22061af6782c67c259d29a96620aa94717dfa725fc946452d3aa63f2e28b8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2e60ba579b1102e62294742b74f8e61420bb9aa2c402df4ab140e4707d642f7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9933de8afe0d9130001fcf9310f68abc3b271652a1b5845bab04dbd4b102a40b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee34607108e9771f4020711248144f2a5b2752a47d24eae37065952a44744c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ed87d2754b95d9bb878d1aafecd4b58ec9d9f25afbda9c9159a444e9975619e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce548847023ca5339f41e1f31ff2ca31a5dcfda6021d4ca7586c529b236e264f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c30a9ec946ae64d4378e524f9673824bd21cce429885ce6b58d5e59f8a514d1e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8562c09aaa62f38d399642a1605077b2a48de70d2e830ad36410d11601e2593","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26d51f6386f167c2537ae614f05ae7c26c11ad2063d405c1e7d33260f31e9360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81bc74942e3b4b69bbe2bca54e12e7b5697fd2d9a9ea07c4bdedd89c288dcaeb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7eb11f8aed55292a0786ab508a2318c68e83946b8e235f5962799e2760c9edb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:942e16d259689aa951fd19d22cc022865036523b79adb3cde7ca90344f8d0911","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79ef7e9bc69a50a3d8c1d6383252de025539dfdefb732a4c435c22acd755be55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f4f2bda645b6bc40d4817755bb3198777ad8f3a9b40daf1bb787e86ec25495a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:228b85a61088f759a7cf51c0c7f0400f1dcbabf04502bb09f54d578e5fcbcb89","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af71b562c81f1b39837792cbf0603912e23adde874935f86e54c3f17f2d8dcfe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b870e2d4d4bd2a036a22a129d24758f933c889d5f6470d9f4efd633fcbb5e3c2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e811aaed4c0bddd3fe8fc328a9bf91ea8f469d704219cf45f4c96a735608174b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e73ab7356731c6c217cb98bf6be0e3a7d5a67903bb337a732f5807aeeb2f970","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f0301bd3209f850b10601f1cf87733cdb203b46629ae443591ba07e45874ac65","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb31af950689a3680c65b8feb71f2bf0ee77c2087b9690fbe67d93a9d99e03ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2b5610a871be9c411eff2b88996f63aa771be44722783c428ab2105570bd237","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11b324feb44479d06f73fdbbbc5f77f6627175fbad49a4a6d9530769885366bc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7204b2dd3cc11ae64b834cfb10b7b73158270bcc7066614dfc856ff0316d2659","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2505982070054c8fefe751d0cfa133e162b233456314e2a3cde782de69653993","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea1d3585c24a32f4a385e05a823d2c52374837b46cbc3051eb79f167226207e9"]},{"name":"q4n4i3p4b4e6x0b-nps-420-422-m7fqd-4v5cx","ready":true,"containerRuntimeVersion":"cri-o://1.35.4-2.rhaos4.22.git6479b5e.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:759db300fd0e0258b2179aa48de5575baa2e33e6d0b6efebdf36cde5647baa40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dda3faf2a605b97850ab295e289f3b5e6c624b5a30bc6f799b8c36bde0504319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05d60b3807f2273ac84ebfceee6f8342bf2eb7d14f87d4ba346e3911a6a48d62","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98457921e71a329fbdefb60e1bbd3de80b145489221fbe9150eb90e1abed9786","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13c7f68070664781a1a338f565f1f24425a764f28557280a0bbf9da51a5172a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b519469b498b28347d0a504ef69d2f480cc4431e50d3972933a86147d215d1e3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:65f250c7337f5765db69690b4530e31a6bf4050d9140c4c85527e7c26e77b7cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:836532b41550b915235e842a4c2215953e78dac14357e6566cfdaa72eafe5dfd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b0b1a151b65d0e34ee0dcd8eb8137b41d3e11a0e2a9042774152be53a47fa2f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d22bf965b7f6ac63e87483965fe644012abd6ef3e46703f90b53b09ffa66c40a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a75e4d50927d4591b602dbb803130d388e3b36123b8770be67e890826639c364","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b434ceae2127cb4872490f8a906e0e3da22307ec413997683dbd518955b9fbcd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ed76865cd6513d71c46844d3027f3f7b161835b360d2e69b59e297019ecd3fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9e0bd9b962e5326507ec7fd7df5e397e1e88c9f7117bdace555e9eb6cf90447","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0edb65f15d6600020bf0056db95a453688976a6a259251afabd0bd04c18c678a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8806ad37ac4897064437a6a466579fbb65e50d3c06253cb1a56e125455b01cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:236da75c5998805911c6789b80ac704875a79b0abd57a11485218d399ae24be6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d4102507f29eb70d722c8478eb243a4b1913a74f937ed4c1c91f1df5d124ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:801b5a471e7d6c506eea3f249c0f771e2becd4c1a33c03f8355f249a4f7d5f6b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f454167c3e9bea08e362726c4ca4f9f11c8c1c9e8ccfb0612496689f109c1b5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:810098f0663c4a402b3a335cc995a91dac1c4a822b2762f874ba2591fc73f76e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf233f0f2e47df07d74e53d391de580e8db76a58bbda92919feda304a0360c2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8441db4cbe6043f6a4ec0206d37cce75679dbd655310345033d1bce3975760ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cbd95a1f4bc47193aaba2ee9866fc5fd8eab5a1e6fdce8e4b343aa8c49ed6ae5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9c967279d60eeeba741ce8274a719ec4f669aad4eca6eca260cd650eeca9a14","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef9a9ac340cda9a1ea2e571dd741b9df56989d937bf020071efbde3d0333a9f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f8b08e39afa2f499c26c7b286274c4a6430db49fbb87e48eaea935c96de23ff","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98b5f93d5ab68a886c325add352d6675a1fd037db44763c270026e9bd66c28f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cb360c71a907253defcb8fcef0cc29002f23bf6b60926027a6a686749ab809f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c3db74d37a5044a0bbb0c1ef15c7124524891d7af741a98c4477c6d8a5858cd0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5143315a1b4121c00a8240b51db968df01d8c33e49497a3bb94879d218bf64d9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df2cf7a02f8cb20d384ebd144a90c1e45a49b8e33cc81cf9c06dc31f926f0ddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abc8265bad76d67cbb8b6c5c3b674ca1d80180611b2bc053024237aa85367e1b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be44c06ff946fafc0a137cb020af2509b3f2d1f7d8c8b081697ec3483df622d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3ee649c0980af383a375ff9b8287f6f81617ab04eab7b21cea2310ec880070c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dac62b77d6871d5263cf3fc1847466343ce68d80bea6abfb4ba9d70a26b980f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b2fe398a7fe679582f8a653eb139ab8f7b1658ce79daa7d25c8d4aded5dfb9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac6aeac333c22ad2b6c42a40a87067e0acd93d9c45cb4af925c52b87229a7b67","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71d5cf9c12963df3b4a478bc77110df9795a7970d9e74174d4cbf540d89a2b37","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a80616a75bec0e677d4594bf5a7122c792abdacf6e87cd166bb6d2ea616d4af2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1e827b8ef69b1f0a71733c3e646344ccb5fb7e5af146016fd2197ef1c3b560d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ae8602190ff0312684494fb8c5f2aa28c1621837a91e2e765f6e85229cc8c118","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ce5a7e16f173cf9fa5a6810bb784324074d758ccf57f26d47ace826346b7a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5ac3a253cc1e80575f9340f5221edb6b0084f9cf26efea6f582c6e6efd674b4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0dbb2aa398c609b7d717e3df90d57ca64d83211f8303481c3df0fb5f7f609901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9e2af850e2160b07ec78fab74be95928ea1848abbf13aa6f9fe6bc66d8272ab1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19aed0b6d293e76c12cab014c1f37860745beed92b005975cdc0453dc31df3c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82ee0e1d7f7383a28694d18751fa691f1fcfca60a690e081c7b9a30cc4a3b541","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0068453dc71c624460a368872a30785d165d184ad2fcc70159540c35cf564a04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:34853aec39e7fa4e06d003e2f4ad908ca682bca79428755b6c2bd8c4ab3aa5f9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f5c22305bba756410991c00ab3142bb5f24b5aac3ac992afc0d0cf8f66698d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b276ebfe0115876e7e2a1ce1ec99e9762da451104b6d55db5ebc341da8d5f266","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8bcdf087244f707380628f86b8c0c35eeefe870f5107f4693b83632508fbb9d2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96a94e57ccff7cac3eba0a3c64ddca0371624f31b025bccab550633969249361","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a5a30de78b92f1f0f41f1a2188bb3e94318139c9b6e5fab900a002fa766c67b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82f171298a467819ed387a4b1861d3c86d06e417175592bc6e018c756f1f884c","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ada22061af6782c67c259d29a96620aa94717dfa725fc946452d3aa63f2e28b8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2e60ba579b1102e62294742b74f8e61420bb9aa2c402df4ab140e4707d642f7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9933de8afe0d9130001fcf9310f68abc3b271652a1b5845bab04dbd4b102a40b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee34607108e9771f4020711248144f2a5b2752a47d24eae37065952a44744c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ed87d2754b95d9bb878d1aafecd4b58ec9d9f25afbda9c9159a444e9975619e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce548847023ca5339f41e1f31ff2ca31a5dcfda6021d4ca7586c529b236e264f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c30a9ec946ae64d4378e524f9673824bd21cce429885ce6b58d5e59f8a514d1e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8562c09aaa62f38d399642a1605077b2a48de70d2e830ad36410d11601e2593","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26d51f6386f167c2537ae614f05ae7c26c11ad2063d405c1e7d33260f31e9360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81bc74942e3b4b69bbe2bca54e12e7b5697fd2d9a9ea07c4bdedd89c288dcaeb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7eb11f8aed55292a0786ab508a2318c68e83946b8e235f5962799e2760c9edb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:942e16d259689aa951fd19d22cc022865036523b79adb3cde7ca90344f8d0911","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79ef7e9bc69a50a3d8c1d6383252de025539dfdefb732a4c435c22acd755be55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f4f2bda645b6bc40d4817755bb3198777ad8f3a9b40daf1bb787e86ec25495a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:228b85a61088f759a7cf51c0c7f0400f1dcbabf04502bb09f54d578e5fcbcb89","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af71b562c81f1b39837792cbf0603912e23adde874935f86e54c3f17f2d8dcfe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b870e2d4d4bd2a036a22a129d24758f933c889d5f6470d9f4efd633fcbb5e3c2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e811aaed4c0bddd3fe8fc328a9bf91ea8f469d704219cf45f4c96a735608174b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e73ab7356731c6c217cb98bf6be0e3a7d5a67903bb337a732f5807aeeb2f970","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f0301bd3209f850b10601f1cf87733cdb203b46629ae443591ba07e45874ac65","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb31af950689a3680c65b8feb71f2bf0ee77c2087b9690fbe67d93a9d99e03ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2b5610a871be9c411eff2b88996f63aa771be44722783c428ab2105570bd237","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11b324feb44479d06f73fdbbbc5f77f6627175fbad49a4a6d9530769885366bc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7204b2dd3cc11ae64b834cfb10b7b73158270bcc7066614dfc856ff0316d2659","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4b3475443fd6e386e1b4819b7d3e1b483d885a36ae3259276ae406e8777f4dc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8200a0b5e1733af9376f87fd9b6606b2c53bae0aa489e5385dd7b296c3d1aac7","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"q4n4i3p4b4e6x0b-nps-420-422-m7fqd-x2ftv","ready":true,"containerRuntimeVersion":"cri-o://1.35.4-2.rhaos4.22.git6479b5e.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:759db300fd0e0258b2179aa48de5575baa2e33e6d0b6efebdf36cde5647baa40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dda3faf2a605b97850ab295e289f3b5e6c624b5a30bc6f799b8c36bde0504319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05d60b3807f2273ac84ebfceee6f8342bf2eb7d14f87d4ba346e3911a6a48d62","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98457921e71a329fbdefb60e1bbd3de80b145489221fbe9150eb90e1abed9786","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13c7f68070664781a1a338f565f1f24425a764f28557280a0bbf9da51a5172a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b519469b498b28347d0a504ef69d2f480cc4431e50d3972933a86147d215d1e3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:65f250c7337f5765db69690b4530e31a6bf4050d9140c4c85527e7c26e77b7cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:836532b41550b915235e842a4c2215953e78dac14357e6566cfdaa72eafe5dfd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b0b1a151b65d0e34ee0dcd8eb8137b41d3e11a0e2a9042774152be53a47fa2f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d22bf965b7f6ac63e87483965fe644012abd6ef3e46703f90b53b09ffa66c40a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a75e4d50927d4591b602dbb803130d388e3b36123b8770be67e890826639c364","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b434ceae2127cb4872490f8a906e0e3da22307ec413997683dbd518955b9fbcd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ed76865cd6513d71c46844d3027f3f7b161835b360d2e69b59e297019ecd3fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9e0bd9b962e5326507ec7fd7df5e397e1e88c9f7117bdace555e9eb6cf90447","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0edb65f15d6600020bf0056db95a453688976a6a259251afabd0bd04c18c678a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8806ad37ac4897064437a6a466579fbb65e50d3c06253cb1a56e125455b01cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:236da75c5998805911c6789b80ac704875a79b0abd57a11485218d399ae24be6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d4102507f29eb70d722c8478eb243a4b1913a74f937ed4c1c91f1df5d124ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:801b5a471e7d6c506eea3f249c0f771e2becd4c1a33c03f8355f249a4f7d5f6b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f454167c3e9bea08e362726c4ca4f9f11c8c1c9e8ccfb0612496689f109c1b5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:810098f0663c4a402b3a335cc995a91dac1c4a822b2762f874ba2591fc73f76e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf233f0f2e47df07d74e53d391de580e8db76a58bbda92919feda304a0360c2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8441db4cbe6043f6a4ec0206d37cce75679dbd655310345033d1bce3975760ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cbd95a1f4bc47193aaba2ee9866fc5fd8eab5a1e6fdce8e4b343aa8c49ed6ae5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9c967279d60eeeba741ce8274a719ec4f669aad4eca6eca260cd650eeca9a14","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef9a9ac340cda9a1ea2e571dd741b9df56989d937bf020071efbde3d0333a9f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f8b08e39afa2f499c26c7b286274c4a6430db49fbb87e48eaea935c96de23ff","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98b5f93d5ab68a886c325add352d6675a1fd037db44763c270026e9bd66c28f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cb360c71a907253defcb8fcef0cc29002f23bf6b60926027a6a686749ab809f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c3db74d37a5044a0bbb0c1ef15c7124524891d7af741a98c4477c6d8a5858cd0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5143315a1b4121c00a8240b51db968df01d8c33e49497a3bb94879d218bf64d9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df2cf7a02f8cb20d384ebd144a90c1e45a49b8e33cc81cf9c06dc31f926f0ddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:381e6243dfb2a2014086ba2984bb884ef33e5be85556e1234cdde6b39036a2c3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9c8a464c8461bba5985c4bee1812ea3e9cbca1ed4ad4a56d955ca13c5c1b36c3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e69e35933e03b9564191cb011ca691b26a221f7e4de53a548d87db8f72e19b8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be754db16286616aedfb46c0a35fa0b33bd6203936f18f668a740c1c6937727f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cdc9bccd17fd7b70880f7e1e412dcfcaa8f00c68769953c0d50242d5f724a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8378d17bf5983a203e2eb01fa04c28ddddd4cfcc0500941e723dc274599e752","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abc8265bad76d67cbb8b6c5c3b674ca1d80180611b2bc053024237aa85367e1b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be44c06ff946fafc0a137cb020af2509b3f2d1f7d8c8b081697ec3483df622d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3ee649c0980af383a375ff9b8287f6f81617ab04eab7b21cea2310ec880070c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dac62b77d6871d5263cf3fc1847466343ce68d80bea6abfb4ba9d70a26b980f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b2fe398a7fe679582f8a653eb139ab8f7b1658ce79daa7d25c8d4aded5dfb9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac6aeac333c22ad2b6c42a40a87067e0acd93d9c45cb4af925c52b87229a7b67","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca6925ab1bbfe57f393f22f75bc0dc38835c4d9d9f81b21136a1a17c505fc6d5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f73e8c279ee2f4e095871b201b1348e918884a5e5fff5cc904093da3c95a1826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71d5cf9c12963df3b4a478bc77110df9795a7970d9e74174d4cbf540d89a2b37","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a80616a75bec0e677d4594bf5a7122c792abdacf6e87cd166bb6d2ea616d4af2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1e827b8ef69b1f0a71733c3e646344ccb5fb7e5af146016fd2197ef1c3b560d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ae8602190ff0312684494fb8c5f2aa28c1621837a91e2e765f6e85229cc8c118","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ce5a7e16f173cf9fa5a6810bb784324074d758ccf57f26d47ace826346b7a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5ac3a253cc1e80575f9340f5221edb6b0084f9cf26efea6f582c6e6efd674b4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b1db94e0479c927d9e4054fcffea2730896dcdc2a0daa10ba507a0139af3842","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16434ab04642837914b2f8bba0ec37254bb7537ebebf3cced37d90b5a6072a19","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19aed0b6d293e76c12cab014c1f37860745beed92b005975cdc0453dc31df3c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82ee0e1d7f7383a28694d18751fa691f1fcfca60a690e081c7b9a30cc4a3b541","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0068453dc71c624460a368872a30785d165d184ad2fcc70159540c35cf564a04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:34853aec39e7fa4e06d003e2f4ad908ca682bca79428755b6c2bd8c4ab3aa5f9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f5c22305bba756410991c00ab3142bb5f24b5aac3ac992afc0d0cf8f66698d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b276ebfe0115876e7e2a1ce1ec99e9762da451104b6d55db5ebc341da8d5f266","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8bcdf087244f707380628f86b8c0c35eeefe870f5107f4693b83632508fbb9d2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96a94e57ccff7cac3eba0a3c64ddca0371624f31b025bccab550633969249361","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:404f2407b90debc8d3a9a6d0b4e837a2319fbd770a2baac09ff42a5e17da1f96","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:822ea04c7980a97b95c28c728057990a66da72a31b3e8c1c111a20f80a86af78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a5a30de78b92f1f0f41f1a2188bb3e94318139c9b6e5fab900a002fa766c67b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82f171298a467819ed387a4b1861d3c86d06e417175592bc6e018c756f1f884c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17eabe2bd670af62415deda51066ba77ea598e2bb95ef524f40e474bdc35263f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5c0f173ea02a36e71d0880c4d41a8dd25965110a153ed705d426355e1effce37","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ada22061af6782c67c259d29a96620aa94717dfa725fc946452d3aa63f2e28b8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2e60ba579b1102e62294742b74f8e61420bb9aa2c402df4ab140e4707d642f7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9933de8afe0d9130001fcf9310f68abc3b271652a1b5845bab04dbd4b102a40b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee34607108e9771f4020711248144f2a5b2752a47d24eae37065952a44744c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ed87d2754b95d9bb878d1aafecd4b58ec9d9f25afbda9c9159a444e9975619e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce548847023ca5339f41e1f31ff2ca31a5dcfda6021d4ca7586c529b236e264f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c30a9ec946ae64d4378e524f9673824bd21cce429885ce6b58d5e59f8a514d1e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8562c09aaa62f38d399642a1605077b2a48de70d2e830ad36410d11601e2593","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26d51f6386f167c2537ae614f05ae7c26c11ad2063d405c1e7d33260f31e9360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81bc74942e3b4b69bbe2bca54e12e7b5697fd2d9a9ea07c4bdedd89c288dcaeb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7eb11f8aed55292a0786ab508a2318c68e83946b8e235f5962799e2760c9edb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:942e16d259689aa951fd19d22cc022865036523b79adb3cde7ca90344f8d0911","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70ee5bbfb23f3b6ba023569bca38d36a735b888db07cef92522dcee14a53bbf5","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:94b38a7bc9b7306facc5355b3287d4560981be14608f472a0af1cae4d8306706","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:228b85a61088f759a7cf51c0c7f0400f1dcbabf04502bb09f54d578e5fcbcb89","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af71b562c81f1b39837792cbf0603912e23adde874935f86e54c3f17f2d8dcfe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b870e2d4d4bd2a036a22a129d24758f933c889d5f6470d9f4efd633fcbb5e3c2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e811aaed4c0bddd3fe8fc328a9bf91ea8f469d704219cf45f4c96a735608174b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e73ab7356731c6c217cb98bf6be0e3a7d5a67903bb337a732f5807aeeb2f970","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f0301bd3209f850b10601f1cf87733cdb203b46629ae443591ba07e45874ac65","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb31af950689a3680c65b8feb71f2bf0ee77c2087b9690fbe67d93a9d99e03ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2b5610a871be9c411eff2b88996f63aa771be44722783c428ab2105570bd237","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11b324feb44479d06f73fdbbbc5f77f6627175fbad49a4a6d9530769885366bc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7204b2dd3cc11ae64b834cfb10b7b73158270bcc7066614dfc856ff0316d2659","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4b3475443fd6e386e1b4819b7d3e1b483d885a36ae3259276ae406e8777f4dc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8200a0b5e1733af9376f87fd9b6606b2c53bae0aa489e5385dd7b296c3d1aac7"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:517]: Timed out after 2700.005s.
Expected success, but got an error:
    <*errors.errorString | 0xc00149cc90>: 
    node pool upgrade verification failed: q4n4i3p4b4e6x0b-nps-420-422-gxph5-9xn25 (version 4.20 not in same minor as expected 4.22.1); nodes=[{"name":"q4n4i3p4b4e6x0b-nps-420-422-gxph5-9xn25","ready":true,"containerRuntimeVersion":"cri-o://1.33.12-3.rhaos4.20.git64c6a00.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05d60b3807f2273ac84ebfceee6f8342bf2eb7d14f87d4ba346e3911a6a48d62","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98457921e71a329fbdefb60e1bbd3de80b145489221fbe9150eb90e1abed9786","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13c7f68070664781a1a338f565f1f24425a764f28557280a0bbf9da51a5172a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b519469b498b28347d0a504ef69d2f480cc4431e50d3972933a86147d215d1e3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:65f250c7337f5765db69690b4530e31a6bf4050d9140c4c85527e7c26e77b7cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:836532b41550b915235e842a4c2215953e78dac14357e6566cfdaa72eafe5dfd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:40234292b7d4c28149b5493acbaeb608ac59326dbeb53d3cfdcdc90f2c98928b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad892d1fda138cbefcd7b1bb79c47897fb31c0a09429d53001aa777d9af7b8d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:918ffd5c62760ee30d3a6874c5b98cf8a82e2c471fb5a9df501909c4d671e9fb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa2835a5b193048ba6b300577088213cc90621c32cf6d8ebfe3d7e2fcee12dbc","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ed76865cd6513d71c46844d3027f3f7b161835b360d2e69b59e297019ecd3fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9e0bd9b962e5326507ec7fd7df5e397e1e88c9f7117bdace555e9eb6cf90447","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0edb65f15d6600020bf0056db95a453688976a6a259251afabd0bd04c18c678a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8806ad37ac4897064437a6a466579fbb65e50d3c06253cb1a56e125455b01cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:236da75c5998805911c6789b80ac704875a79b0abd57a11485218d399ae24be6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d4102507f29eb70d722c8478eb243a4b1913a74f937ed4c1c91f1df5d124ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:801b5a471e7d6c506eea3f249c0f771e2becd4c1a33c03f8355f249a4f7d5f6b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f454167c3e9bea08e362726c4ca4f9f11c8c1c9e8ccfb0612496689f109c1b5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:810098f0663c4a402b3a335cc995a91dac1c4a822b2762f874ba2591fc73f76e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf233f0f2e47df07d74e53d391de580e8db76a58bbda92919feda304a0360c2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8441db4cbe6043f6a4ec0206d37cce75679dbd655310345033d1bce3975760ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cbd95a1f4bc47193aaba2ee9866fc5fd8eab5a1e6fdce8e4b343aa8c49ed6ae5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9c967279d60eeeba741ce8274a719ec4f669aad4eca6eca260cd650eeca9a14","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef9a9ac340cda9a1ea2e571dd741b9df56989d937bf020071efbde3d0333a9f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f8b08e39afa2f499c26c7b286274c4a6430db49fbb87e48eaea935c96de23ff","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98b5f93d5ab68a886c325add352d6675a1fd037db44763c270026e9bd66c28f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cb360c71a907253defcb8fcef0cc29002f23bf6b60926027a6a686749ab809f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c3db74d37a5044a0bbb0c1ef15c7124524891d7af741a98c4477c6d8a5858cd0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5143315a1b4121c00a8240b51db968df01d8c33e49497a3bb94879d218bf64d9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df2cf7a02f8cb20d384ebd144a90c1e45a49b8e33cc81cf9c06dc31f926f0ddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abc8265bad76d67cbb8b6c5c3b674ca1d80180611b2bc053024237aa85367e1b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be44c06ff946fafc0a137cb020af2509b3f2d1f7d8c8b081697ec3483df622d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3ee649c0980af383a375ff9b8287f6f81617ab04eab7b21cea2310ec880070c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dac62b77d6871d5263cf3fc1847466343ce68d80bea6abfb4ba9d70a26b980f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b2fe398a7fe679582f8a653eb139ab8f7b1658ce79daa7d25c8d4aded5dfb9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac6aeac333c22ad2b6c42a40a87067e0acd93d9c45cb4af925c52b87229a7b67","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71d5cf9c12963df3b4a478bc77110df9795a7970d9e74174d4cbf540d89a2b37","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a80616a75bec0e677d4594bf5a7122c792abdacf6e87cd166bb6d2ea616d4af2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1e827b8ef69b1f0a71733c3e646344ccb5fb7e5af146016fd2197ef1c3b560d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ae8602190ff0312684494fb8c5f2aa28c1621837a91e2e765f6e85229cc8c118","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ce5a7e16f173cf9fa5a6810bb784324074d758ccf57f26d47ace826346b7a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5ac3a253cc1e80575f9340f5221edb6b0084f9cf26efea6f582c6e6efd674b4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0dbb2aa398c609b7d717e3df90d57ca64d83211f8303481c3df0fb5f7f609901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9e2af850e2160b07ec78fab74be95928ea1848abbf13aa6f9fe6bc66d8272ab1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:117a9ef195508484c302bb9e03e4e9330f97029ea97eabbe6d00a48fd6640aed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b62893ea3da92e93050dd06e3a47ccb74e18178999b4217f2a4b812b7f7e5126","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19aed0b6d293e76c12cab014c1f37860745beed92b005975cdc0453dc31df3c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82ee0e1d7f7383a28694d18751fa691f1fcfca60a690e081c7b9a30cc4a3b541","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0068453dc71c624460a368872a30785d165d184ad2fcc70159540c35cf564a04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:34853aec39e7fa4e06d003e2f4ad908ca682bca79428755b6c2bd8c4ab3aa5f9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f5c22305bba756410991c00ab3142bb5f24b5aac3ac992afc0d0cf8f66698d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b276ebfe0115876e7e2a1ce1ec99e9762da451104b6d55db5ebc341da8d5f266","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8bcdf087244f707380628f86b8c0c35eeefe870f5107f4693b83632508fbb9d2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96a94e57ccff7cac3eba0a3c64ddca0371624f31b025bccab550633969249361","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a5a30de78b92f1f0f41f1a2188bb3e94318139c9b6e5fab900a002fa766c67b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82f171298a467819ed387a4b1861d3c86d06e417175592bc6e018c756f1f884c","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ae36b4ba7f205b81ef595f0c59c7a1548520d29309598a37328f40708449f2e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd4f32783510721d8e1a61884428d56f44b175c8786498f13f6d554f969f1e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ada22061af6782c67c259d29a96620aa94717dfa725fc946452d3aa63f2e28b8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2e60ba579b1102e62294742b74f8e61420bb9aa2c402df4ab140e4707d642f7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9933de8afe0d9130001fcf9310f68abc3b271652a1b5845bab04dbd4b102a40b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee34607108e9771f4020711248144f2a5b2752a47d24eae37065952a44744c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ed87d2754b95d9bb878d1aafecd4b58ec9d9f25afbda9c9159a444e9975619e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce548847023ca5339f41e1f31ff2ca31a5dcfda6021d4ca7586c529b236e264f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c30a9ec946ae64d4378e524f9673824bd21cce429885ce6b58d5e59f8a514d1e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8562c09aaa62f38d399642a1605077b2a48de70d2e830ad36410d11601e2593","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26d51f6386f167c2537ae614f05ae7c26c11ad2063d405c1e7d33260f31e9360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81bc74942e3b4b69bbe2bca54e12e7b5697fd2d9a9ea07c4bdedd89c288dcaeb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7eb11f8aed55292a0786ab508a2318c68e83946b8e235f5962799e2760c9edb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:942e16d259689aa951fd19d22cc022865036523b79adb3cde7ca90344f8d0911","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79ef7e9bc69a50a3d8c1d6383252de025539dfdefb732a4c435c22acd755be55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f4f2bda645b6bc40d4817755bb3198777ad8f3a9b40daf1bb787e86ec25495a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:228b85a61088f759a7cf51c0c7f0400f1dcbabf04502bb09f54d578e5fcbcb89","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af71b562c81f1b39837792cbf0603912e23adde874935f86e54c3f17f2d8dcfe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b870e2d4d4bd2a036a22a129d24758f933c889d5f6470d9f4efd633fcbb5e3c2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e811aaed4c0bddd3fe8fc328a9bf91ea8f469d704219cf45f4c96a735608174b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e73ab7356731c6c217cb98bf6be0e3a7d5a67903bb337a732f5807aeeb2f970","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f0301bd3209f850b10601f1cf87733cdb203b46629ae443591ba07e45874ac65","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb31af950689a3680c65b8feb71f2bf0ee77c2087b9690fbe67d93a9d99e03ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2b5610a871be9c411eff2b88996f63aa771be44722783c428ab2105570bd237","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11b324feb44479d06f73fdbbbc5f77f6627175fbad49a4a6d9530769885366bc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7204b2dd3cc11ae64b834cfb10b7b73158270bcc7066614dfc856ff0316d2659","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2505982070054c8fefe751d0cfa133e162b233456314e2a3cde782de69653993","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ea1d3585c24a32f4a385e05a823d2c52374837b46cbc3051eb79f167226207e9"]},{"name":"q4n4i3p4b4e6x0b-nps-420-422-m7fqd-4v5cx","ready":true,"containerRuntimeVersion":"cri-o://1.35.4-2.rhaos4.22.git6479b5e.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:759db300fd0e0258b2179aa48de5575baa2e33e6d0b6efebdf36cde5647baa40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dda3faf2a605b97850ab295e289f3b5e6c624b5a30bc6f799b8c36bde0504319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05d60b3807f2273ac84ebfceee6f8342bf2eb7d14f87d4ba346e3911a6a48d62","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98457921e71a329fbdefb60e1bbd3de80b145489221fbe9150eb90e1abed9786","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13c7f68070664781a1a338f565f1f24425a764f28557280a0bbf9da51a5172a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b519469b498b28347d0a504ef69d2f480cc4431e50d3972933a86147d215d1e3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:65f250c7337f5765db69690b4530e31a6bf4050d9140c4c85527e7c26e77b7cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:836532b41550b915235e842a4c2215953e78dac14357e6566cfdaa72eafe5dfd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b0b1a151b65d0e34ee0dcd8eb8137b41d3e11a0e2a9042774152be53a47fa2f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d22bf965b7f6ac63e87483965fe644012abd6ef3e46703f90b53b09ffa66c40a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a75e4d50927d4591b602dbb803130d388e3b36123b8770be67e890826639c364","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b434ceae2127cb4872490f8a906e0e3da22307ec413997683dbd518955b9fbcd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ed76865cd6513d71c46844d3027f3f7b161835b360d2e69b59e297019ecd3fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9e0bd9b962e5326507ec7fd7df5e397e1e88c9f7117bdace555e9eb6cf90447","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0edb65f15d6600020bf0056db95a453688976a6a259251afabd0bd04c18c678a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8806ad37ac4897064437a6a466579fbb65e50d3c06253cb1a56e125455b01cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:236da75c5998805911c6789b80ac704875a79b0abd57a11485218d399ae24be6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d4102507f29eb70d722c8478eb243a4b1913a74f937ed4c1c91f1df5d124ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:801b5a471e7d6c506eea3f249c0f771e2becd4c1a33c03f8355f249a4f7d5f6b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f454167c3e9bea08e362726c4ca4f9f11c8c1c9e8ccfb0612496689f109c1b5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:810098f0663c4a402b3a335cc995a91dac1c4a822b2762f874ba2591fc73f76e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf233f0f2e47df07d74e53d391de580e8db76a58bbda92919feda304a0360c2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8441db4cbe6043f6a4ec0206d37cce75679dbd655310345033d1bce3975760ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cbd95a1f4bc47193aaba2ee9866fc5fd8eab5a1e6fdce8e4b343aa8c49ed6ae5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9c967279d60eeeba741ce8274a719ec4f669aad4eca6eca260cd650eeca9a14","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef9a9ac340cda9a1ea2e571dd741b9df56989d937bf020071efbde3d0333a9f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f8b08e39afa2f499c26c7b286274c4a6430db49fbb87e48eaea935c96de23ff","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98b5f93d5ab68a886c325add352d6675a1fd037db44763c270026e9bd66c28f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cb360c71a907253defcb8fcef0cc29002f23bf6b60926027a6a686749ab809f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c3db74d37a5044a0bbb0c1ef15c7124524891d7af741a98c4477c6d8a5858cd0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5143315a1b4121c00a8240b51db968df01d8c33e49497a3bb94879d218bf64d9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df2cf7a02f8cb20d384ebd144a90c1e45a49b8e33cc81cf9c06dc31f926f0ddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abc8265bad76d67cbb8b6c5c3b674ca1d80180611b2bc053024237aa85367e1b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be44c06ff946fafc0a137cb020af2509b3f2d1f7d8c8b081697ec3483df622d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3ee649c0980af383a375ff9b8287f6f81617ab04eab7b21cea2310ec880070c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dac62b77d6871d5263cf3fc1847466343ce68d80bea6abfb4ba9d70a26b980f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b2fe398a7fe679582f8a653eb139ab8f7b1658ce79daa7d25c8d4aded5dfb9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac6aeac333c22ad2b6c42a40a87067e0acd93d9c45cb4af925c52b87229a7b67","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71d5cf9c12963df3b4a478bc77110df9795a7970d9e74174d4cbf540d89a2b37","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a80616a75bec0e677d4594bf5a7122c792abdacf6e87cd166bb6d2ea616d4af2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1e827b8ef69b1f0a71733c3e646344ccb5fb7e5af146016fd2197ef1c3b560d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ae8602190ff0312684494fb8c5f2aa28c1621837a91e2e765f6e85229cc8c118","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ce5a7e16f173cf9fa5a6810bb784324074d758ccf57f26d47ace826346b7a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5ac3a253cc1e80575f9340f5221edb6b0084f9cf26efea6f582c6e6efd674b4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0dbb2aa398c609b7d717e3df90d57ca64d83211f8303481c3df0fb5f7f609901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9e2af850e2160b07ec78fab74be95928ea1848abbf13aa6f9fe6bc66d8272ab1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19aed0b6d293e76c12cab014c1f37860745beed92b005975cdc0453dc31df3c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82ee0e1d7f7383a28694d18751fa691f1fcfca60a690e081c7b9a30cc4a3b541","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0068453dc71c624460a368872a30785d165d184ad2fcc70159540c35cf564a04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:34853aec39e7fa4e06d003e2f4ad908ca682bca79428755b6c2bd8c4ab3aa5f9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f5c22305bba756410991c00ab3142bb5f24b5aac3ac992afc0d0cf8f66698d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b276ebfe0115876e7e2a1ce1ec99e9762da451104b6d55db5ebc341da8d5f266","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8bcdf087244f707380628f86b8c0c35eeefe870f5107f4693b83632508fbb9d2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96a94e57ccff7cac3eba0a3c64ddca0371624f31b025bccab550633969249361","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a5a30de78b92f1f0f41f1a2188bb3e94318139c9b6e5fab900a002fa766c67b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82f171298a467819ed387a4b1861d3c86d06e417175592bc6e018c756f1f884c","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ada22061af6782c67c259d29a96620aa94717dfa725fc946452d3aa63f2e28b8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2e60ba579b1102e62294742b74f8e61420bb9aa2c402df4ab140e4707d642f7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9933de8afe0d9130001fcf9310f68abc3b271652a1b5845bab04dbd4b102a40b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee34607108e9771f4020711248144f2a5b2752a47d24eae37065952a44744c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ed87d2754b95d9bb878d1aafecd4b58ec9d9f25afbda9c9159a444e9975619e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce548847023ca5339f41e1f31ff2ca31a5dcfda6021d4ca7586c529b236e264f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c30a9ec946ae64d4378e524f9673824bd21cce429885ce6b58d5e59f8a514d1e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8562c09aaa62f38d399642a1605077b2a48de70d2e830ad36410d11601e2593","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26d51f6386f167c2537ae614f05ae7c26c11ad2063d405c1e7d33260f31e9360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81bc74942e3b4b69bbe2bca54e12e7b5697fd2d9a9ea07c4bdedd89c288dcaeb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7eb11f8aed55292a0786ab508a2318c68e83946b8e235f5962799e2760c9edb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:942e16d259689aa951fd19d22cc022865036523b79adb3cde7ca90344f8d0911","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79ef7e9bc69a50a3d8c1d6383252de025539dfdefb732a4c435c22acd755be55","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f4f2bda645b6bc40d4817755bb3198777ad8f3a9b40daf1bb787e86ec25495a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:228b85a61088f759a7cf51c0c7f0400f1dcbabf04502bb09f54d578e5fcbcb89","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af71b562c81f1b39837792cbf0603912e23adde874935f86e54c3f17f2d8dcfe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b870e2d4d4bd2a036a22a129d24758f933c889d5f6470d9f4efd633fcbb5e3c2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e811aaed4c0bddd3fe8fc328a9bf91ea8f469d704219cf45f4c96a735608174b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e73ab7356731c6c217cb98bf6be0e3a7d5a67903bb337a732f5807aeeb2f970","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f0301bd3209f850b10601f1cf87733cdb203b46629ae443591ba07e45874ac65","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb31af950689a3680c65b8feb71f2bf0ee77c2087b9690fbe67d93a9d99e03ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2b5610a871be9c411eff2b88996f63aa771be44722783c428ab2105570bd237","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11b324feb44479d06f73fdbbbc5f77f6627175fbad49a4a6d9530769885366bc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7204b2dd3cc11ae64b834cfb10b7b73158270bcc7066614dfc856ff0316d2659","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4b3475443fd6e386e1b4819b7d3e1b483d885a36ae3259276ae406e8777f4dc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8200a0b5e1733af9376f87fd9b6606b2c53bae0aa489e5385dd7b296c3d1aac7","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"q4n4i3p4b4e6x0b-nps-420-422-m7fqd-x2ftv","ready":true,"containerRuntimeVersion":"cri-o://1.35.4-2.rhaos4.22.git6479b5e.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:759db300fd0e0258b2179aa48de5575baa2e33e6d0b6efebdf36cde5647baa40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dda3faf2a605b97850ab295e289f3b5e6c624b5a30bc6f799b8c36bde0504319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05d60b3807f2273ac84ebfceee6f8342bf2eb7d14f87d4ba346e3911a6a48d62","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98457921e71a329fbdefb60e1bbd3de80b145489221fbe9150eb90e1abed9786","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13c7f68070664781a1a338f565f1f24425a764f28557280a0bbf9da51a5172a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b519469b498b28347d0a504ef69d2f480cc4431e50d3972933a86147d215d1e3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:65f250c7337f5765db69690b4530e31a6bf4050d9140c4c85527e7c26e77b7cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:836532b41550b915235e842a4c2215953e78dac14357e6566cfdaa72eafe5dfd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b0b1a151b65d0e34ee0dcd8eb8137b41d3e11a0e2a9042774152be53a47fa2f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d22bf965b7f6ac63e87483965fe644012abd6ef3e46703f90b53b09ffa66c40a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a75e4d50927d4591b602dbb803130d388e3b36123b8770be67e890826639c364","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b434ceae2127cb4872490f8a906e0e3da22307ec413997683dbd518955b9fbcd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ed76865cd6513d71c46844d3027f3f7b161835b360d2e69b59e297019ecd3fa","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c9e0bd9b962e5326507ec7fd7df5e397e1e88c9f7117bdace555e9eb6cf90447","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0edb65f15d6600020bf0056db95a453688976a6a259251afabd0bd04c18c678a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8806ad37ac4897064437a6a466579fbb65e50d3c06253cb1a56e125455b01cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:236da75c5998805911c6789b80ac704875a79b0abd57a11485218d399ae24be6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d4102507f29eb70d722c8478eb243a4b1913a74f937ed4c1c91f1df5d124ce","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:801b5a471e7d6c506eea3f249c0f771e2becd4c1a33c03f8355f249a4f7d5f6b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f454167c3e9bea08e362726c4ca4f9f11c8c1c9e8ccfb0612496689f109c1b5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:810098f0663c4a402b3a335cc995a91dac1c4a822b2762f874ba2591fc73f76e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf233f0f2e47df07d74e53d391de580e8db76a58bbda92919feda304a0360c2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8441db4cbe6043f6a4ec0206d37cce75679dbd655310345033d1bce3975760ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cbd95a1f4bc47193aaba2ee9866fc5fd8eab5a1e6fdce8e4b343aa8c49ed6ae5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9c967279d60eeeba741ce8274a719ec4f669aad4eca6eca260cd650eeca9a14","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef9a9ac340cda9a1ea2e571dd741b9df56989d937bf020071efbde3d0333a9f3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f8b08e39afa2f499c26c7b286274c4a6430db49fbb87e48eaea935c96de23ff","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98b5f93d5ab68a886c325add352d6675a1fd037db44763c270026e9bd66c28f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cb360c71a907253defcb8fcef0cc29002f23bf6b60926027a6a686749ab809f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c3db74d37a5044a0bbb0c1ef15c7124524891d7af741a98c4477c6d8a5858cd0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5143315a1b4121c00a8240b51db968df01d8c33e49497a3bb94879d218bf64d9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df2cf7a02f8cb20d384ebd144a90c1e45a49b8e33cc81cf9c06dc31f926f0ddc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:381e6243dfb2a2014086ba2984bb884ef33e5be85556e1234cdde6b39036a2c3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9c8a464c8461bba5985c4bee1812ea3e9cbca1ed4ad4a56d955ca13c5c1b36c3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e69e35933e03b9564191cb011ca691b26a221f7e4de53a548d87db8f72e19b8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be754db16286616aedfb46c0a35fa0b33bd6203936f18f668a740c1c6937727f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5cdc9bccd17fd7b70880f7e1e412dcfcaa8f00c68769953c0d50242d5f724a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a8378d17bf5983a203e2eb01fa04c28ddddd4cfcc0500941e723dc274599e752","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abc8265bad76d67cbb8b6c5c3b674ca1d80180611b2bc053024237aa85367e1b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be44c06ff946fafc0a137cb020af2509b3f2d1f7d8c8b081697ec3483df622d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3ee649c0980af383a375ff9b8287f6f81617ab04eab7b21cea2310ec880070c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dac62b77d6871d5263cf3fc1847466343ce68d80bea6abfb4ba9d70a26b980f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b2fe398a7fe679582f8a653eb139ab8f7b1658ce79daa7d25c8d4aded5dfb9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ac6aeac333c22ad2b6c42a40a87067e0acd93d9c45cb4af925c52b87229a7b67","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ca6925ab1bbfe57f393f22f75bc0dc38835c4d9d9f81b21136a1a17c505fc6d5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f73e8c279ee2f4e095871b201b1348e918884a5e5fff5cc904093da3c95a1826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71d5cf9c12963df3b4a478bc77110df9795a7970d9e74174d4cbf540d89a2b37","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a80616a75bec0e677d4594bf5a7122c792abdacf6e87cd166bb6d2ea616d4af2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1e827b8ef69b1f0a71733c3e646344ccb5fb7e5af146016fd2197ef1c3b560d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ae8602190ff0312684494fb8c5f2aa28c1621837a91e2e765f6e85229cc8c118","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ce5a7e16f173cf9fa5a6810bb784324074d758ccf57f26d47ace826346b7a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c5ac3a253cc1e80575f9340f5221edb6b0084f9cf26efea6f582c6e6efd674b4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0b1db94e0479c927d9e4054fcffea2730896dcdc2a0daa10ba507a0139af3842","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16434ab04642837914b2f8bba0ec37254bb7537ebebf3cced37d90b5a6072a19","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:19aed0b6d293e76c12cab014c1f37860745beed92b005975cdc0453dc31df3c7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82ee0e1d7f7383a28694d18751fa691f1fcfca60a690e081c7b9a30cc4a3b541","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0068453dc71c624460a368872a30785d165d184ad2fcc70159540c35cf564a04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:34853aec39e7fa4e06d003e2f4ad908ca682bca79428755b6c2bd8c4ab3aa5f9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:35f5c22305bba756410991c00ab3142bb5f24b5aac3ac992afc0d0cf8f66698d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b276ebfe0115876e7e2a1ce1ec99e9762da451104b6d55db5ebc341da8d5f266","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8bcdf087244f707380628f86b8c0c35eeefe870f5107f4693b83632508fbb9d2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:96a94e57ccff7cac3eba0a3c64ddca0371624f31b025bccab550633969249361","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:404f2407b90debc8d3a9a6d0b4e837a2319fbd770a2baac09ff42a5e17da1f96","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:822ea04c7980a97b95c28c728057990a66da72a31b3e8c1c111a20f80a86af78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5a5a30de78b92f1f0f41f1a2188bb3e94318139c9b6e5fab900a002fa766c67b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82f171298a467819ed387a4b1861d3c86d06e417175592bc6e018c756f1f884c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17eabe2bd670af62415deda51066ba77ea598e2bb95ef524f40e474bdc35263f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5c0f173ea02a36e71d0880c4d41a8dd25965110a153ed705d426355e1effce37","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ada22061af6782c67c259d29a96620aa94717dfa725fc946452d3aa63f2e28b8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2e60ba579b1102e62294742b74f8e61420bb9aa2c402df4ab140e4707d642f7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9933de8afe0d9130001fcf9310f68abc3b271652a1b5845bab04dbd4b102a40b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee34607108e9771f4020711248144f2a5b2752a47d24eae37065952a44744c6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ed87d2754b95d9bb878d1aafecd4b58ec9d9f25afbda9c9159a444e9975619e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce548847023ca5339f41e1f31ff2ca31a5dcfda6021d4ca7586c529b236e264f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c30a9ec946ae64d4378e524f9673824bd21cce429885ce6b58d5e59f8a514d1e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f8562c09aaa62f38d399642a1605077b2a48de70d2e830ad36410d11601e2593","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26d51f6386f167c2537ae614f05ae7c26c11ad2063d405c1e7d33260f31e9360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81bc74942e3b4b69bbe2bca54e12e7b5697fd2d9a9ea07c4bdedd89c288dcaeb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7eb11f8aed55292a0786ab508a2318c68e83946b8e235f5962799e2760c9edb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:942e16d259689aa951fd19d22cc022865036523b79adb3cde7ca90344f8d0911","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70ee5bbfb23f3b6ba023569bca38d36a735b888db07cef92522dcee14a53bbf5","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:94b38a7bc9b7306facc5355b3287d4560981be14608f472a0af1cae4d8306706","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:228b85a61088f759a7cf51c0c7f0400f1dcbabf04502bb09f54d578e5fcbcb89","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af71b562c81f1b39837792cbf0603912e23adde874935f86e54c3f17f2d8dcfe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b870e2d4d4bd2a036a22a129d24758f933c889d5f6470d9f4efd633fcbb5e3c2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e811aaed4c0bddd3fe8fc328a9bf91ea8f469d704219cf45f4c96a735608174b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e73ab7356731c6c217cb98bf6be0e3a7d5a67903bb337a732f5807aeeb2f970","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f0301bd3209f850b10601f1cf87733cdb203b46629ae443591ba07e45874ac65","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb31af950689a3680c65b8feb71f2bf0ee77c2087b9690fbe67d93a9d99e03ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f2b5610a871be9c411eff2b88996f63aa771be44722783c428ab2105570bd237","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11b324feb44479d06f73fdbbbc5f77f6627175fbad49a4a6d9530769885366bc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7204b2dd3cc11ae64b834cfb10b7b73158270bcc7066614dfc856ff0316d2659","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4b3475443fd6e386e1b4819b7d3e1b483d885a36ae3259276ae406e8777f4dc4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8200a0b5e1733af9376f87fd9b6606b2c53bae0aa489e5385dd7b296c3d1aac7"]}]
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067198813889630208