No failure patterns match your search.
Environment: PROD
Failure patterns: 9 · Failures matched: 23 · Total runs: 14 · Runs affected: 8
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Deserializaion Error: no output from commandfull failure pattern: Deserializaion Error: no output from command Signal: NoiseSignal: Noise — source deserialization/no-output error | e2e | 2 | 14.29%2 of 14 job runs affected | NoiseSignal: Noise — source deserialization/no-output error | Show trend detailsMar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 3 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (3)/opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539718 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:224
"ts"="2026-03-26 15:41:58.528583" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating resource group @ 03/26/26 15:55:00.664
"ts"="2026-03-26 15:55:00.664343" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
STEP: creating cluster parameters at control plane version @ 03/26/26 15:55:04.464
"ts"="2026-03-26 15:55:04.464563" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:55:04.464576" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:55:04.464
STEP: creating the HCP cluster with version 4.21 @ 03/26/26 15:56:17.712
"ts"="2026-03-26 15:56:17.712675" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-58q68d" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 03/26/26 16:07:15.153
"ts"="2026-03-26 16:07:15.153711" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 16:07:15.153724" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 03/26/26 16:16:41.008
STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 03/26/26 16:17:18.424
STEP: capturing node release images before upgrade @ 03/26/26 16:17:18.558
STEP: triggering nodepool upgrade to version 4.21.5 @ 03/26/26 16:17:18.709
STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 03/26/26 16:18:14.022
STEP: verifying node pool GET still reflects the new version @ 03/26/26 16:46:24.898
"ts"="2026-03-26 16:46:25.999092" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 16:46:25.999204" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-58q68d-4t4lnx" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 16:46:26.755016" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-58q68d-4t4lnx" "resourceGroup"=""
"ts"="2026-03-26 16:46:28.395939" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:46:28.396047" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:46:28.396124" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 16:46:28.397295" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-124"
"ts"="2026-03-26 16:46:28.397316" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 16:46:28.601756" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-124" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 16:46:28.749306" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-124" "deployment"="msi-deployment-124"
"ts"="2026-03-26 16:46:28.873805" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk" "deployment"="pooledNonMsiScopedAssignments-ad26qwpi75bc2"
"ts"="2026-03-26 16:46:29.618362" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk" "deployment"="customer-infra-np-version-upgrade-cluster-58q68d-4t4lnx"
"ts"="2026-03-26 16:46:30.395419" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 16:46:30.395563" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 16:46:30.395689" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 17:06:14.835068" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 17:06:14.835095" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
Command Error: signal: killed
Deserializaion Error: no output from command
/opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539718 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:224
"ts"="2026-03-26 15:41:58.528583" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating resource group @ 03/26/26 15:55:00.664
"ts"="2026-03-26 15:55:00.664343" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
STEP: creating cluster parameters at control plane version @ 03/26/26 15:55:04.464
"ts"="2026-03-26 15:55:04.464563" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:55:04.464576" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:55:04.464
STEP: creating the HCP cluster with version 4.21 @ 03/26/26 15:56:17.712
"ts"="2026-03-26 15:56:17.712675" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-58q68d" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 03/26/26 16:07:15.153
"ts"="2026-03-26 16:07:15.153711" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 16:07:15.153724" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 03/26/26 16:16:41.008
STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 03/26/26 16:17:18.424
STEP: capturing node release images before upgrade @ 03/26/26 16:17:18.558
STEP: triggering nodepool upgrade to version 4.21.5 @ 03/26/26 16:17:18.709
STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 03/26/26 16:18:14.022
STEP: verifying node pool GET still reflects the new version @ 03/26/26 16:46:24.898
"ts"="2026-03-26 16:46:25.999092" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 16:46:25.999204" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-58q68d-4t4lnx" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 16:46:26.755016" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-58q68d-4t4lnx" "resourceGroup"=""
"ts"="2026-03-26 16:46:28.395939" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:46:28.396047" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:46:28.396124" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 16:46:28.397295" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-124"
"ts"="2026-03-26 16:46:28.397316" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 16:46:28.601756" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-124" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 16:46:28.749306" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-124" "deployment"="msi-deployment-124"
"ts"="2026-03-26 16:46:28.873805" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk" "deployment"="pooledNonMsiScopedAssignments-ad26qwpi75bc2"
"ts"="2026-03-26 16:46:29.618362" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk" "deployment"="customer-infra-np-version-upgrade-cluster-58q68d-4t4lnx"
"ts"="2026-03-26 16:46:30.395419" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 16:46:30.395563" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 16:46:30.395689" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 17:06:14.835068" "level"=0 "msg"="no left behind managed resource groups found" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
"ts"="2026-03-26 17:06:14.835095" "level"=0 "msg"="deleting resource group" "resourceGroup"="rg-np-version-upgrade-58q68d-6q4tvk"
Command Error: signal: killed
Deserializaion Error: no output from commandRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539959 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
STEP: creating a resource group @ 03/26/26 15:46:02.033
"ts"="2026-03-26 15:46:02.033400" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-7xsgnf"
STEP: creating cluster parameters @ 03/26/26 15:46:06.196
"ts"="2026-03-26 15:46:06.196672" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:46:06.196689" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:46:06.196
STEP: creating the HCP cluster @ 03/26/26 15:46:48.119
"ts"="2026-03-26 15:46:48.120027" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-7xsgnf"
STEP: getting credentials @ 03/26/26 15:58:00.443
STEP: creating the node pool with initial labels and taints @ 03/26/26 15:58:47.704
"ts"="2026-03-26 15:58:47.704646" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 15:58:47.704659" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: waiting for nodes to be ready @ 03/26/26 16:09:40.126
STEP: verifying initial labels are present on nodes @ 03/26/26 16:09:40.961
STEP: verifying initial taints are present on nodes @ 03/26/26 16:09:41.073
STEP: updating nodepool with new taints and scaling up @ 03/26/26 16:09:41.073
STEP: verifying nodes are scaled to the expected count @ 03/26/26 16:17:48.889
STEP: verifying new taint is present on newly created node @ 03/26/26 16:17:49.423
STEP: updating nodepool with a new label and scaling up @ 03/26/26 16:17:49.528
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:225 @ 03/26/26 17:02:49.529
"ts"="2026-03-26 17:02:49.529703" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 17:02:49.529828" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-dvwx88" "resourceGroup"="rg-np-labels-taints-7xsgnf"
"ts"="2026-03-26 17:02:50.522417" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-dvwx88" "resourceGroup"=""
"ts"="2026-03-26 17:02:52.051693" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-03-26 17:02:52.051782" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-03-26 17:02:52.051836" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 17:02:52.052935" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-25"
"ts"="2026-03-26 17:02:52.052955" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-7xsgnf"
"ts"="2026-03-26 17:02:52.249459" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 17:02:52.393486" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="msi-deployment-25"
"ts"="2026-03-26 17:02:52.538837" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-7xsgnf" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
"ts"="2026-03-26 17:02:53.235631" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-7xsgnf" "deployment"="customer-infra-cluster-np-labels-taints-dvwx88"
"ts"="2026-03-26 17:02:53.762868" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 17:02:53.762999" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 17:02:53.763130" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-7xsgnf"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539959 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should be able to update node pool labels and taints [PreLaunchSetup:None, Importance:Medium, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:39
STEP: creating a resource group @ 03/26/26 15:46:02.033
"ts"="2026-03-26 15:46:02.033400" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-labels-taints-7xsgnf"
STEP: creating cluster parameters @ 03/26/26 15:46:06.196
"ts"="2026-03-26 15:46:06.196672" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:46:06.196689" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:46:06.196
STEP: creating the HCP cluster @ 03/26/26 15:46:48.119
"ts"="2026-03-26 15:46:48.120027" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="cluster-np-labels-taints" "resourceGroup"="rg-np-labels-taints-7xsgnf"
STEP: getting credentials @ 03/26/26 15:58:00.443
STEP: creating the node pool with initial labels and taints @ 03/26/26 15:58:47.704
"ts"="2026-03-26 15:58:47.704646" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 15:58:47.704659" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: waiting for nodes to be ready @ 03/26/26 16:09:40.126
STEP: verifying initial labels are present on nodes @ 03/26/26 16:09:40.961
STEP: verifying initial taints are present on nodes @ 03/26/26 16:09:41.073
STEP: updating nodepool with new taints and scaling up @ 03/26/26 16:09:41.073
STEP: verifying nodes are scaled to the expected count @ 03/26/26 16:17:48.889
STEP: verifying new taint is present on newly created node @ 03/26/26 16:17:49.423
STEP: updating nodepool with a new label and scaling up @ 03/26/26 16:17:49.528
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:225 @ 03/26/26 17:02:49.529
"ts"="2026-03-26 17:02:49.529703" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 17:02:49.529828" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-cluster-np-labels-taints-dvwx88" "resourceGroup"="rg-np-labels-taints-7xsgnf"
"ts"="2026-03-26 17:02:50.522417" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-cluster-np-labels-taints-dvwx88" "resourceGroup"=""
"ts"="2026-03-26 17:02:52.051693" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-03-26 17:02:52.051782" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-11d9063b8cbda82463f6839234612e7ec58e097fdac2676470a369504689ddfe.yaml.gz"
"ts"="2026-03-26 17:02:52.051836" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 17:02:52.052935" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-25"
"ts"="2026-03-26 17:02:52.052955" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-labels-taints-7xsgnf"
"ts"="2026-03-26 17:02:52.249459" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 17:02:52.393486" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-25" "deployment"="msi-deployment-25"
"ts"="2026-03-26 17:02:52.538837" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-7xsgnf" "deployment"="pooledNonMsiScopedAssignments-r6cuakilyx67k"
"ts"="2026-03-26 17:02:53.235631" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-labels-taints-7xsgnf" "deployment"="customer-infra-cluster-np-labels-taints-dvwx88"
"ts"="2026-03-26 17:02:53.762868" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 17:02:53.762999" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 17:02:53.763130" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-labels-taints-7xsgnf"
Command Error: signal: killed
Deserializaion Error: no output from commandRunning Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539962 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:224
"ts"="2026-03-26 15:46:02.082189" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating resource group @ 03/26/26 15:55:02.492
"ts"="2026-03-26 15:55:02.492194" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
STEP: creating cluster parameters at control plane version @ 03/26/26 15:55:06.122
"ts"="2026-03-26 15:55:06.123025" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:55:06.123040" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:55:06.123
STEP: creating the HCP cluster with version 4.21 @ 03/26/26 15:55:57.394
"ts"="2026-03-26 15:55:57.394964" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-25sqpq" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 03/26/26 16:07:52.506
"ts"="2026-03-26 16:07:52.506148" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 16:07:52.506162" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 03/26/26 16:19:27.978
STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 03/26/26 16:20:38.475
STEP: capturing node release images before upgrade @ 03/26/26 16:20:38.598
STEP: triggering nodepool upgrade to version 4.21.5 @ 03/26/26 16:20:38.808
STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 03/26/26 16:21:44.788
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214 @ 03/26/26 16:51:44.79
"ts"="2026-03-26 16:51:44.790837" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 16:51:44.790942" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-25sqpq-kkwrph" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
"ts"="2026-03-26 16:51:45.965086" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-25sqpq-kkwrph" "resourceGroup"=""
"ts"="2026-03-26 16:51:48.588028" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:51:48.588123" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:51:48.588183" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 16:51:48.589294" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-49"
"ts"="2026-03-26 16:51:48.589305" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
"ts"="2026-03-26 16:51:48.745161" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-49" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 16:51:48.756385" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz" "deployment"="pooledNonMsiScopedAssignments-7f6zxb5wg4rdw"
"ts"="2026-03-26 16:51:48.889592" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-49" "deployment"="msi-deployment-49"
"ts"="2026-03-26 16:51:49.447325" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz" "deployment"="customer-infra-np-version-upgrade-cluster-25sqpq-kkwrph"
"ts"="2026-03-26 16:51:50.126755" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 16:51:50.126870" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 16:51:50.127003" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
Command Error: signal: killed
Deserializaion Error: no output from command
Running Suite: - /opt/app-root/src/github.com/Azure/ARO-HCP
============================================================
Random Seed: 1774539962 - will randomize all specs
Will run 1 of 1 specs
------------------------------
Customer should upgrade a nodepool from 4.20.z to 4.21.zLatest [PreLaunchSetup:None, Importance:Critical, Positivity:Positive, ARO-HCP-RP-API-Compatible]
/opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:224
"ts"="2026-03-26 15:46:02.082189" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating resource group @ 03/26/26 15:55:02.492
"ts"="2026-03-26 15:55:02.492194" "level"=0 "msg"="creating resource group" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
STEP: creating cluster parameters at control plane version @ 03/26/26 15:55:06.122
"ts"="2026-03-26 15:55:06.123025" "level"=0 "msg"="Using OpenShift control plane version" "version"="4.20"
"ts"="2026-03-26 15:55:06.123040" "level"=0 "msg"="Using OpenShift channel group" "channelGroup"="stable"
STEP: creating customer resources @ 03/26/26 15:55:06.123
STEP: creating the HCP cluster with version 4.21 @ 03/26/26 15:55:57.394
"ts"="2026-03-26 15:55:57.394964" "level"=0 "msg"="Starting HCP cluster creation" "clusterName"="np-version-upgrade-cluster-25sqpq" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
STEP: creating nodepool with version 4.20.16 (behind control plane; upgrade path validated via Cincinnati) @ 03/26/26 16:07:52.506
"ts"="2026-03-26 16:07:52.506148" "level"=0 "msg"="Using OpenShift node pool version" "version"="4.20.15"
"ts"="2026-03-26 16:07:52.506162" "level"=0 "msg"="Using OpenShift node pool channel group" "channelGroup"="stable"
STEP: getting admin credentials and lowest control plane version from OpenShift version history @ 03/26/26 16:19:27.978
STEP: computing nodepool desired version from Cincinnati (lowest <= lowest control plane version with upgrade path from nodepool) @ 03/26/26 16:20:38.475
STEP: capturing node release images before upgrade @ 03/26/26 16:20:38.598
STEP: triggering nodepool upgrade to version 4.21.5 @ 03/26/26 16:20:38.808
STEP: verifying nodes are ready, updated to expected version, and release images differ from pre-upgrade @ 03/26/26 16:21:44.788
[FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214 @ 03/26/26 16:51:44.79
"ts"="2026-03-26 16:51:44.790837" "level"=0 "msg"="Commiting timing metadata."
"ts"="2026-03-26 16:51:44.790942" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-np-version-upgrade-cluster-25sqpq-kkwrph" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
"ts"="2026-03-26 16:51:45.965086" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-np-version-upgrade-cluster-25sqpq-kkwrph" "resourceGroup"=""
"ts"="2026-03-26 16:51:48.588028" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:51:48.588123" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-d75bd3b573d86e38666f5431fc1f04dc7901ed57b6ee602c711bd58b56307a0b.yaml.gz"
"ts"="2026-03-26 16:51:48.588183" "level"=0 "msg"="collecting debug info"
"ts"="2026-03-26 16:51:48.589294" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-49"
"ts"="2026-03-26 16:51:48.589305" "level"=0 "msg"="collecting deployments" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
"ts"="2026-03-26 16:51:48.745161" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-49" "deployment"="pooledMsiScopedAssignments"
"ts"="2026-03-26 16:51:48.756385" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz" "deployment"="pooledNonMsiScopedAssignments-7f6zxb5wg4rdw"
"ts"="2026-03-26 16:51:48.889592" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-49" "deployment"="msi-deployment-49"
"ts"="2026-03-26 16:51:49.447325" "level"=0 "msg"="collecting operations" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz" "deployment"="customer-infra-np-version-upgrade-cluster-25sqpq-kkwrph"
"ts"="2026-03-26 16:51:50.126755" "level"=0 "msg"="finished collecting debug info"
"ts"="2026-03-26 16:51:50.126870" "level"=0 "msg"="deleting created resources"
"ts"="2026-03-26 16:51:50.127003" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="rg-np-version-upgrade-25sqpq-rxgqtz"
Command Error: signal: killed
Deserializaion Error: no output from commandContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 4 | 28.57%4 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsMar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 5 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapErrors | 0xc000dbc090>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-vg7dpc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-vg7dpc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0011c6190>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapErrors | 0xc000dbc090>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-vg7dpc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-vg7dpc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc0011c6190>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-vg7dpc",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapErrors | 0xc0011193b0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-65cxhw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-65cxhw, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-65cxhw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-65cxhw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000fab760>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-65cxhw",
},
<context.deadlineExceededError>{},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:134]: Unexpected error:
<*fmt.wrapErrors | 0xc0011193b0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-65cxhw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-65cxhw, error: context deadline exceeded
{
msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-65cxhw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-65cxhw, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000fab760>{
s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-65cxhw",
},
<context.deadlineExceededError>{},
],
}
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
<*fmt.wrapErrors | 0xc0012ec390>:
failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-rjfzzc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc, error: context deadline exceeded
{
msg: "failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc, error: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-rjfzzc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ee3970>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc",
},
<*fmt.wrapErrors | 0xc0012ec360>{
msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-rjfzzc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ee3970>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc",
},
<context.deadlineExceededError>{},
],
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
<*fmt.wrapErrors | 0xc0012ec390>:
failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-rjfzzc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc, error: context deadline exceeded
{
msg: "failed to create HCP cluster np-autoscale-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc, error: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-rjfzzc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ee3970>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc",
},
<*fmt.wrapErrors | 0xc0012ec360>{
msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-rjfzzc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc, error: context deadline exceeded",
errs: [
<*errors.errorString | 0xc000ee3970>{
s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-rjfzzc",
},
<context.deadlineExceededError>{},
],
},
],
}
occurredContributing tests (4)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||
ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscri...full failure pattern: ERROR CODE: DeploymentFailed; detail code CannotChangePrivateLinkConnectionOnPrivateEndpoint; detail message Cannot change the private link connection on private endpoint /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 14.29%2 of 14 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 4 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc0014ce600>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-khck8x" in resourcegroup="customer-rg-2djx9g" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-khck8x\" in resourcegroup=\"customer-rg-2djx9g\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0014ce5e0>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-khck8x\" in resourcegroup=\"customer-rg-2djx9g\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001454ed0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:96]: Unexpected error:
<*fmt.wrapError | 0xc0014ce600>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster2-khck8x" in resourcegroup="customer-rg-2djx9g" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster2-khck8x\" in resourcegroup=\"customer-rg-2djx9g\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0014ce5e0>{
msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster2-khck8x\" in resourcegroup=\"customer-rg-2djx9g\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-2djx9g/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster2-khck8x/operationStatuses/08584270478335360636\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-2djx9g/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001454ed0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 200,
Proto: "HTTP/2.0",
ProtoMajor: 2,
ProtoMinor...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc0008cc0e0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-6jgfhl" in resourcegroup="rg-cluster-nsg-subnet-reuse-2l4lzn" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-6jgfhl\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-2l4lzn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008cc0c0>{
msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-6jgfhl\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-2l4lzn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e040f0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:109]: Unexpected error:
<*fmt.wrapError | 0xc0008cc0e0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-subnet-reuse-6jgfhl" in resourcegroup="rg-cluster-nsg-subnet-reuse-2l4lzn" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-cluster-subnet-reuse-6jgfhl\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-2l4lzn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc0008cc0c0>{
msg: "failed waiting for deployment \"customer-infra-cluster-subnet-reuse-6jgfhl\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-2l4lzn\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Resources/deployments/customer-infra-cluster-subnet-reuse-6jgfhl/operationStatuses/08584270510574480059\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-2l4lzn/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc000e040f0>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
<*fmt.wrapError | 0xc001568640>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-4jws2q" in resourcegroup="rg-cluster-nsg-subnet-reuse-czjmtq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-4jws2q\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-czjmtq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001568620>{
msg: "failed waiting for deployment \"customer-infra-basic-cluster-4jws2q\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-czjmtq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001484f00>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 2...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
<*fmt.wrapError | 0xc001568640>:
failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-4jws2q" in resourcegroup="rg-cluster-nsg-subnet-reuse-czjmtq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "BadRequest",
"message": "{\r\n \"error\": {\r\n \"code\": \"CannotChangePrivateLinkConnectionOnPrivateEndpoint\",\r\n \"message\": \"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\",\r\n \"details\": []\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
{
msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-4jws2q\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-czjmtq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*fmt.wrapError | 0xc001568620>{
msg: "failed waiting for deployment \"customer-infra-basic-cluster-4jws2q\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-czjmtq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-4jws2q/operationStatuses/08584270510594480916\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"BadRequest\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"CannotChangePrivateLinkConnectionOnPrivateEndpoint\\\",\\r\\n \\\"message\\\": \\\"Cannot change the private link connection on private endpoint /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint. Please ensure you are not updating the details of existing private link connection: '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-czjmtq/providers/Microsoft.Network/privateEndpoints/kv-private-endpoint/privateLinkServiceConnections/kv-private-endpoint'. That is not allowed.\\\",\\r\\n \\\"details\\\": []\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n",
err: <*exported.ResponseError | 0xc001484f00>{
ErrorCode: "DeploymentFailed",
StatusCode: 200,
RawResponse: {
Status: "200 OK",
StatusCode: 2...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||
Process did not finish before 2h0m0s timeoutfull failure pattern: Process did not finish before 2h0m0s timeout Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | unknown | 2 | 14.29%2 of 14 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 2 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1){"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:169","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 2h0m0s timeout","severity":"error","time":"2026-03-26T17:07:37Z"}Contributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||
node pool <nodepool> verification failed: <node> (version 4full failure pattern: node pool <nodepool> verification failed: <node> (version 4 Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 2 | 14.29%2 of 14 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 2 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.004s.
Expected success, but got an error:
<*errors.errorString | 0xc0002fe2c0>:
node pool upgrade verification failed: q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648"]},{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-qhrz9-4hzfn","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-qhrz9-psqk4","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.004s.
Expected success, but got an error:
<*errors.errorString | 0xc0002fe2c0>:
node pool upgrade verification failed: q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a9f74fb0da7fb9f7d70c01953cb70fdda6a22bb9182bc5358f5fd6d957f4f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ef42cb417552fe1e38eedc30e92f55c8c50aae1aa1b0b7094ab56118efe0d96c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a3c906f4908c692697ae58891ef463ee62a23d4356bf00786947cc1dc02e4d53","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ec94607aa5fce47591b1ed1939e86a0e1c43cc8ff439f42662c9ed32e414595e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:321163a14934467a86e358bbb3115fbe104ad814bcaf7423f259eb82150da703","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2906e0869191da12130ed79a57221aae735ef026d0f909534c7cbfbca131128","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d4067116d68c1aabb5ee21c4addce05f6849f104d7a7f87f5f1fcef7990a3aee","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e199029f436d7247195d0c5b22909cb874ac67bd87d84c597a83ef7bd0d87ab0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648"]},{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-qhrz9-4hzfn","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]},{"name":"q7z8q4o0i9x2i6b-npupgrade-4-20-qhrz9-psqk4","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"q7z8q4o0i9x2i6b-npupgrade-4-20-h7rgh-jmgdc\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7f1a2243123d8b98d100e0ecc5c189db3d4bf89f7edee3e5f0331fb48f8803c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:be6ed9469bb1acf2658aaf55b812e5334e308b4169b858b0bacf54ce6f4e2ca7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-outputfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.004s.
Expected success, but got an error:
<*errors.errorString | 0xc0002ff7a0>:
node pool upgrade verification failed: t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdf391b97b311ecf8f0e5be97024014ad8b264d7785313081aa1290eaf0bc100","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fadd21095ba14bd0d033a74eacbd06f7e40a03b9acbaa846a85a5dc16d73ca00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aadc8e5d5141737eec06abcb21778ea74ceca2076ed31a2b1277f332d837d2f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9eb3bea4e623795326319ebb95624f9b2a0905aac10ced9e74260931aa33ab1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6afe7e59c25f2df8e44971d20bf90820ff5c66abac524e656b2f84121032126","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3172c4a59d36cfbe0567767264b230cb7b1bc355e8753cbf14da088987601d06","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3115b2610585407ab0742648cfbe39c72f57482889f0e778f5ac6fdc482217b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462"]},{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-rpdl4-5c7rp","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36"]},{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-rpdl4-ws7nw","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:214]: Timed out after 1800.004s.
Expected success, but got an error:
<*errors.errorString | 0xc0002ff7a0>:
node pool upgrade verification failed: t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx (version 4.20 not in same minor as expected 4.21.5); nodes=[{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx","ready":true,"containerRuntimeVersion":"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1652a6b3078719d56385dddda165a7e63168c645aa45ca408e7b12e682dcbdab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7b0e43c637df9024af6222a689d54ba25e55f3a2a663202636b1b137b5a47a8c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9b87244518245c63d1982182abe22775a2e207511cb328bc5c7b48e435b8da7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16dfb8b96a665be805bc1000677823aee946d93bfff0d3c567f2a24ea28a537a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2d76c5a5fca4f9ef200178f0c8d63557a6b351fd55f4be46ca1793de5a83400","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5dc767cd6267e006720af8cef541e23bc64d94ef9b1bf3a8b44b1fda5d9d8871","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6afb374cc68bb9bf8199044b29de3348f9cc8cb18acc336a0c7762ab18666927","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6cf0a0701c4429a00483da429c3fa7d0f175d8e1602b2413860e8fc0c9f0e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:95c0990b3f768502d64d40d4330e411543c7207bfb7eb2eb44fd31c0b39edc3d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78c23f5bccb6e82ade8428c8886ddc0291faec496590b2e6b5375ac6c0476dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9ff94628410a702de0cb3e1a715917829b36c3def524dcc0850ac34a811f8f6f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2bcca887297fc1168fa43a6c1acb7077bf198258fe2e0ef9379e11c418919c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d7376a666810d810549b4e270d9a2fd6bcfd8799370ff52476a4f799cd4e287d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:602dc9f92b7a8e84e6eb998191263d186d1d10d378eeef0845435715bee812dc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f7953a8e77c5bb0efaa670ba2188050740eba0e1f6979248832c05cb886fa5d0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c4fdcf25b69610f918eac662369ed457ee400b3d5483e8fcbecd44a97c220302","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b369532d212af90d1232bcf0a84bc2616106733f4eae8d4aaf69fad595e1d67","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bdf391b97b311ecf8f0e5be97024014ad8b264d7785313081aa1290eaf0bc100","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fadd21095ba14bd0d033a74eacbd06f7e40a03b9acbaa846a85a5dc16d73ca00","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aadc8e5d5141737eec06abcb21778ea74ceca2076ed31a2b1277f332d837d2f5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d9eb3bea4e623795326319ebb95624f9b2a0905aac10ced9e74260931aa33ab1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6afe7e59c25f2df8e44971d20bf90820ff5c66abac524e656b2f84121032126","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3172c4a59d36cfbe0567767264b230cb7b1bc355e8753cbf14da088987601d06","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b3115b2610585407ab0742648cfbe39c72f57482889f0e778f5ac6fdc482217b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462"]},{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-rpdl4-5c7rp","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c24b0dd9206d3e91907593da7cad54d881306b287ad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ade6dfb5406ef3a4264dea95d455298f54e8ef4d458d9ee3e94c33d48676a9b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ba2d747627cdfa2d7d36728dac5444865455d422df60f33943a45cef211badb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12aec97ddeb527881acf5949bea2b227baa1dbfccf30679e8d57ff0b82e42707","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f88c407e2bdf86c7339ced2ae92bd716d5f74cee0999d41c5e3b26d47cfe2648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:20a3e59566dd25058fff22e18569067fe6a12d8ac5ac62fa6508e0cee44d7897","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a405e40da35645d318b95c414f68536e3d3e2db22394ca9bf9738e666336e462","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:12448aad6e9050c2b2445035be818a490cef502d39b73dbd51cc0f5ea7d67d40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17a5d6b121e2bb5b8656a7a427824c2af9254ee0bd1f5cec30072b7c564eae7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:66640a3f7bd65090345552611701a73e74bc44897b9d29e43b11f73b96598ac0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:adcbb731ad586feb2c21dbc4993b26a564bbcb26efffb3f94f9da38e27002ea8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2f26472542da4f89a2ac34a93fd438d385419dda16811bdfa7e679e0115abe56","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e19f841ff1628bef075f117708faad18b1f525e6629542bb2db3031acac948fe","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6263faa8ef7759ec21f26df08314c5d64e52d967b49961cb9404c54532c03365","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b37b3692dc74deb0b12b592173a09fb371a0a87210861f14239c70f247431294","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5bd409efe77c6a126418be2c098dde9431c1a290cec885a7e37c0c8ddc2337a8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24f5d0b962f9b381df0f8d90d4c31616a95623fbca70de0a2da621aba6bd19f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a07c1b2ef06bbfa3a491b84f7410e24de7cb487f49bbbb3bfd7cb3510c8429a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd3da568f95c65b04a03d96d175d64efcabcb6fe9a454a1658bbfa243b16a0a7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b58ba57c10e2ba03a0d8dba1f3f8faa866e1dca6170725b6909c2cc2b96efa6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f02f34239d2898fd346b775194a7e3f1a26b8aaeeaf3670b67cc5e4021742420","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5f852f6127d27e6fc071afe58d80024370467cb2476f4d9f077f8ebca8600918","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6dd7872d95388063d6606823e4c4340f83a4f6983034f265a4d749f0418e77c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36"]},{"name":"t5v5b3j7c0v5c4l-npupgrade-4-20-rpdl4-ws7nw","ready":true,"containerRuntimeVersion":"cri-o://1.34.5-3.rhaos4.21.gita8af6ea.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6148e980bdc4394e5ee766bac607fc607777a5c1c2dad4c18e243c006121235c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5547fcebf920ead6c1f429068078852ea910912bb1ac2942a5f05f8eea4e968","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:746e5cb3f7701cf0afcbea92f34ada0c4ea060fc2691a59ebd1ed8239a8e61b5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:abd3eaae48639c784a21da52d7da93bd9ba38c938fc5941c9990958fd6b47f5d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3da5a449ace61cb7d002d8e7911c14905c74cf58a296527458d1af0f1abf0a72","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:464c7d227e1c80e413d87d99fdf42fc80d76b3bc8f9281a23b7d59b3b4ea2372","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6ea0f19030ffc52a290e6eeb812ef2426c48a2f1083b1e415e0667998cbb7bf2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cf9f6b6fecd35f94d7cdb57845990602be060e4b548aac8ab6bc1c96e22f9bb2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1c5b548d67e5c56e3297b2f8aa34428e544b5b5ac88bd8d0f465760375c501e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bc7e9750e4647b2ccd9a7c1c0f4cc35f640b36d8ab9d3fa789eb2764afce3d7a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6b085cdad58db8529d99c22ec1ec79fe09ab00ab1c92ee38b29e7875dc0533c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2efd3418ad3517f5db50de37cda00d27bb568554e7cb618ade7c8eebfbc9185","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:24929612f2f0f2d4cd3ac163c04bbf4ac64cc47d3d0eef81db085d00c59058bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:987fe55afa2ee263467ecaf5c058664ad8e55126687f79d28ab5fa030ec17768","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cca646f56344329bd71967e6b044f43543f52f8c6a8986d725610ad92df26989","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d0d866dee27d9169a1e4711a35a4dfbc3ffe9655258a72188282f4c32eaa84a1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3b9f60077c475f2baad0dd51ffc9d469f3a9b54c474bb86a35f93cb9b8fdc734","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bccd3f544da031a055000bcf0a8fa1003dbe9cfa898bf2fb67c4ee3dcb4f4acc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b48b717cea29d8445ea717563e3794b381f6c1545b4458fb915c8baaa279893a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26a47c35b590b15b04a9558c4fe3e34eef0fecfb4ab6e88274b7e9d70678a8d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c7f80064deb88df54bf40b4834fc5ffa9cd85fc18987fc82c0efdf579e228692","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2044d490f2d73d42bb6a114a2011c898034354e0fab1a872a723174b2c8327cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:39a8fe4ceb03e1379951609b8d1ab33c40aa19f7fe24ed7b0f1621950c051cf6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05ebc4de717d2d36e3d7daca025294e6de1181e3c67ebc38028fc0b0ea5bfafa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23c2d9d4c3f769427c52b6b016633476fcf07a25523a0a3c6bcbf2a1b4840826","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4c39dc633bf3560c2397281b74faf9d00500b0738dec6c1aad2cc8ce326be6fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ba77b4a636a254ef48caadc41666eb92b785317472708cf53d4a108cf7af9a78","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b51a4d9d3b7ecad110fb6eb00f71e9e9b854067d79b220df397a58d9fa50eb0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9fe72e4ea5c3ef2600e9005c2ee869f9fbb68e31b569195e351a55ede1df703a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92a731e25801365be0ad0916bec5bf24b2035e443ad1414a1631d1e0473ddf77","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c91b7e9bdf73069bea2bc8ece90ce15c2a91a2bf99d9d970732b9afa233f3c6c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3f7181447dd3d28779323dac4561ca98d33755c19f468fb181d50c5338eca42a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:46427cb447a096a86618be4829484f964f10c220cd4d779e2c54a960c3045b7b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7e0a58aad44d5b09954f8261e3e428c8dfa076eb9c2b6ef46a91d27ac78ac17a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:83344c7d1374cbc2607b4fcb837f094c035b32daf24f8b3ccba2c8df409a6d3c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3739c0c55f7f7ccc84309ae11eb23d3d9e5ad50a68e849a8a8dfda7f2f2bc137","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a20df9dd8819885aa431022c6f5e80b77b4431f268b719d71bec26f10521da0e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:72ae38682f25786aec18625cfed5d7831017d2f0f1304c3320c00ce0b7fb09c1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9540d66e9824b1ba081ad80bb6715e5ccf61b42c29c1ad961c1863efd81f48ca","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:14e983ca10e1033822e1c4be0a1e4a40762951a5d6e4a77bf642891f291d32ae","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:698e13ede97d3e2fbc3d69858b3f2d5985bbdfb3fb0f646452fe0edabbd5eb7f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:273930332d7b481bbca4521cf740507498a2a323a4010cd66e896806b299f85e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f942448d1f3d05bd5534d16433e0961509c35ea1eab6d51c0d2a8b7e16b0ad29","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e6d59867cbd750672bc9ccfb9c69c3d198174cd43dd6d71a0da358b4b766473","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e99cd629414427a7792ccecda36d856dbe549682438f7b0dfb0ee2e7af536dd6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44ceafe5147680d58a5c2d3720932b972e6bc9639faadc19aa6b360af154e1a3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5b3d41bea99611b73a21042f0f416dd3e8293c93086e18193c98b5fc1994734e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c59765325e6f286c067e555850b0d81f2de5e5ee8625ed83cdc2cea96f3a7ddb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cdd07910c008557d42ada95c4e41333704779166ffa8397b0adc73050a2df3ab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e476b4dafff885727b0057259c9df67a68e196f8cd92c21db2e4efe84a5f7a25","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ee6b4bbee84922afa10d46091e1511788478855b20ce2ad7ac803852f06694","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ad039a130515504a258952cb18e67c3329da5182bb0623cc0891e6c15e2113a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c1f86751c05c53394f4fa5375c71b390f0908ea4a0a8dcc1be27d1aaad2b8130","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4e62a2dcb8214491388d4c17f89fe699eb53cdb8039dcba6ab052edc28da3fbd","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eea3f13f48a70c63b12c74b82caa6b1a8869130a43d5dc3c5ce7e518a4bdf4a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cfcb0bd30c537e677ac6856c36c0c923b415f24d944c7aa1386489f511f6491d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f660d525da7c598367787fef3ac27294357524affe4f35c1f552e7f428ca7dab","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e8d51b4b12d970bf561f67d0fcd9c0966a6d49866e6466daa341ee43d95d6e2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dadfaf3cceca2867f77c40b373b6fc80e34b4ba2cb1f05e7178a0cc79eac2d9d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa65d1c7fb8805b767fc4878faea280472f372010efadc66e5641a2e634a80c0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcd01cbde0c58ad690f233a8f9ac9d080ca607fd7c24ad0de5afca564f098687","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:87de118dc29ca1b47d4768ec5ed4441bc1dada3fb2a78d53b1833249baa86dd4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b615eeec97d8429f11b1f8f9d05c7192c01ad293ed26c83b60bf6e31505f2c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:64dbe77583c8b23dc6852fc482ea5b4081490734944ec8b765d5450b3fdb30ed","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9be8f12e3745100401dcf9f7d7e571260b11399f1dd98501208af756d08abd97","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e3661304917ce7f3be49974faacc60c24cbe2833d7b50c8c205c2f3f2467950","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d2afd0719d1a256d032048beeba2700cfe60da2904b2d9fe9c0aae3daf4318d5","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:1af59b7a29432314bde54e8977fa45fa92dc48885efbf0df601418ec0912f472","quay.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:969bdfb00b7b9f7a99eb3892e34622055e6c98f72c74c64c9a60e1b4b881a246","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad0e262327997e8cdc4074a2373e67274d9b8a943aa274a2cd8ce98ffe1db4a0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d6c8c086bd9f73d1cc4a362ce6253914d4f4d8ce009b68aec42db9ff3a6aad36","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0e5fa2595714628c16cf590efd28715bde790d2dc6b3a793b2a38dfb3ffb591a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5acacbb3eaaddf8d2f25e062427edd7f7636fc7a3b6a9fc84519df16fec0e5e0"]}]
{
s: "node pool upgrade verification failed: t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx (version 4.20 not in same minor as expected 4.21.5); nodes=[{\"name\":\"t5v5b3j7c0v5c4l-npupgrade-4-20-7t2vw-lcjbx\",\"ready\":true,\"containerRuntimeVersion\":\"cri-o://1.33.9-3.rhaos4.20.gitb9ac835.el9\",\"releaseImages\":[\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:779115a9d5f9b0373950fdf99b8bb7bffb8a9143ec28c6e1558d1d3d8eec50e4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70dcc3c7d3a572ecacc7c838fd17d07ff21531e50a7ed9227a17c27437151bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f33338fd94d039dc87a8f9c0fa00221c31bc493bedb70075f21a0fc912bfef04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:185a95b130b6a27ad88fcf39803215b7bd998472b1e37225858435f4eb7c3c04\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab5375f4de81dcf4e990e316124f3d9614ee588bc55342fe94b544602883afa\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e08a29d3b206379957f1fb668058b4effe2f4542c2b4e45a3cfafa4b9a728e5\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9aee9be2da68239a2a559e3ca1211eb8988a7f83a4ff82b9bd7c21a436be9c22\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:51e1b5dc9fccbee5b7df78d0e8bbe379f75529232c102c9ed5b1ae134c4b9b4c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:89c8b051a435031bdcc24715379e0003cb67001076c4134f82ad0dae5cef3704\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ecf24d093581fc39574dceec3d50efd4c3146cebf49bb7a5a450b3f8ce740551\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcdb0afaa38bf8deddfcda9abb50886c2a72c54695992d79c15a8413ac2d1796\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cff1d43f1ec9e919965e5cedd2268233c0f7b27b07750bd6fe8a9bf10f65bf4\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bf18d37657cff26bb88be6fbbdaf64ae8f1f090312233b0c50a904a35729c74c\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a45c25e5322761a9268bbaf5bdf3436d4623f42b90cd15942df0e5c22755e3ac\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:23016e42a746a81e3de1f5661cf2673efbe15ad0926b92d3d7c6fa9ac47f7215\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f6ef619d31c4211120b23a7bf3118c9db7329c81b94709df86396d269d6b30e6\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1adc4e7a9f123529e9df17eb1316a667d86ed2f185965c25ed6d32c44563188b\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b963e4b54323ed252d951c924567590ccb4514d8e90b908c0be89f18919ae8cc\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:31e175b644c6627779ca6b9f48d9426760dd449788ad671dd0277cce8fb822e8\",\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:69e134b501fdb83fc21e3e9e28b43d0f9ddf6643e2a87fc952d2ab48739121a9\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0fd69333c121efd68919e731cd3ea01e976e00415bda6e987bf1e949ec224eda\",\"arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af3a4a739d80ee0b0d8a6c...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-outputContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 4 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0015f6390>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-qh8cxr" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc00096b6c0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00096b660>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00096b640>{
msg: "failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0015f6390>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-qh8cxr" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc00096b6c0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00096b660>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc00096b640>{
msg: "failed waiting for hcpCluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-qh8cxr\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001447860>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-58tkbd" in resourcegroup="rg-np-version-upgrade-58tkbd-xfhc4s" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000943000>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000942fc0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000942f80>{
msg: "failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001447860>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-58tkbd" in resourcegroup="rg-np-version-upgrade-58tkbd-xfhc4s" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000943000>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000942fc0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000942f80>{
msg: "failed waiting for hcpCluster=\"np-version-upgrade-cluster-58tkbd\" in resourcegroup=\"rg-np-version-upgrade-58tkbd-xfhc4s\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0012d3cf8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5dhhk8" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000d51ac0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d51aa0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d51a80>{
msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0012d3cf8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5dhhk8" to finish deleting: context canceled
{
errs: [
<*fmt.wrapError | 0xc000d51ac0>{
msg: "failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d51aa0>{
msg: "at least one hcp cluster failed to delete: failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*fmt.wrapError | 0xc000d51a80>{
msg: "failed waiting for hcpCluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5dhhk8\" to finish deleting: context canceled",
err: <*errors.errorString | 0xb2907c0>{
s: "context canceled",
},
},
},
},
],
}
occurred
fail [:0]: A node timeout occurredContributing tests (4)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqqfull failure pattern: found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqq Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 1 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
<*errors.joinError | 0xc0019920d8>:
found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqq
{
errs: [
<*errors.errorString | 0xc0012612b0>{
s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqq",
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:245]: Unexpected error:
<*errors.joinError | 0xc0019920d8>:
found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqq
{
errs: [
<*errors.errorString | 0xc0012612b0>{
s: "found 1 managed resource groups left behind HCP clusters in cluster-listing-zndkqq",
},
],
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>full failure pattern: tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 1 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:213]: Timed out after 120.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:201 with:
kubectl should execute successfully from authorized VM
Unexpected error:
<*errors.errorString | 0xc0009f2770>:
E0326 02:40:03.194264 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.207738 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.221081 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.504134 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.522643 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
Unable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io
{
s: "E0326 02:40:03.194264 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.207738 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.221081 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.504134 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.522643 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nUnable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io",
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:213]: Timed out after 120.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:201 with:
kubectl should execute successfully from authorized VM
Unexpected error:
<*errors.errorString | 0xc0009f2770>:
E0326 02:40:03.194264 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.207738 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.221081 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.504134 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
E0326 02:40:03.522643 2031 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io"
Unable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io
{
s: "E0326 02:40:03.194264 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.207738 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.221081 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.504134 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nE0326 02:40:03.522643 2031 memcache.go:265] \"Unhandled Error\" err=\"couldn't get current server API group list: Get \\\"https://api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io:443/api?timeout=32s\\\": tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io\"\nUnable to connect to the server: tls: failed to verify certificate: x509: certificate is valid for api.d8m5o7d0p5b0o2p.mxxd.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.d8m5o7d0p5b0o2p.o5hw.uksouth.aroapp-hcp.io",
}
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 7.14%1 of 14 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMar 20: 0 · Mar 21: 0 · Mar 22: 0 · Mar 23: 0 · Mar 24: 0 · Mar 25: 0 · Mar 26: 1 | none | |||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: Unexpected error:
<*fmt.wrapError | 0xc000e5a200>:
route host was never found: Get "https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0xc00111a8a0>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io",
Err: <*tls.CertificateVerificationError | 0xc00111a870>{
UnverifiedCertificates: [
{
Raw: [48, 130, 3, 251, 48, 130, 2, 227, 160, 3, 2, 1, 2, 2, 8, 92, 203, 189, 73, 220, 104, 43, 31, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 51, 50, 54, 48, 54, 52, 53, 49, 48, 90, 23, 13, 50, 55, 48, 51, 50, 54, 48, 54, 52, 53, 49, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 249, 228, 29, 76, 100, 40, 200, 92, 162, 72, 142, 129, 115, 47, 12, 145, 183, 72, 240, 1, 5, 92, 53, 47, 31, 202, 10, 174, 62, 48, 18, 31, 123, 117, 140, 213, 164, 154, 238, 253, 232, 211, 79, 197, 28, 90, 200, 94, 224, 0, 70, 240, 119, 180, 27, 69, 87, 250, 57, 250, 182, 205, 211, 32, 26, 243, 182, 5, 144, 142, 58, 253, 89, 73, 124, 131, 53, 242, 32, 215, 113, 115, 161, 138, 70, 204, 131, 156, 64, 119, 133, 189, 247, 73, 99, 55, 100, 228, 240, 153, 178, 98, 32, 168, 45, 26, 44, 226, 43, 23, 138, 243, 25, 89, 222, 29, 108, 66, 218, 157, 9, 246, 107, 237, 142, 13, 120, 2, 51, 162, 36, 40, 113, 174, 6, 24, 142, 141, 102, 87, 120, 195, 146, 75, 183, 146, 189, 224, 113, 188, 226, 122, 101, 188, 79, 120, 110, 96, 64, 136, 125, 35, 111, 198, 186, 195, 167, 74, 193, 60, 173, 53, 229, 146, 21, 60, 205, 130, 8, 88, 103, 27, 128, 125, 81, 34, 171, 43, 156, 241, 130, 164, 77, 16, 73, 90, 75, 165, 203, 168, 125, 253, 236, 167, 195, 124, 242, 13, 239, 113, 109, 161, 225, 227, 92, 142, 4, 22, 72, 42, 37, 19, 217, 72, 151, 205, 233, 36, 126, 107, 7, 87, 232, 186, 158, 130, 205, 56, 29, 44, 197, 167, 114, 125, 3, 48, 242, 51, 4, 177, 248, 136, 183, 247, 237, 2, 3, 1, 0, 1, 163, 130, 1, 33, 48, 130, 1, 29, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 12, 97, 255, 11, 226, 234, 126, 212, 87, 16, 110, 229, 159, 103, 200, 181, 174, 217, 7, 108, 202, 95, 92, 201, 13, 142, 12, 237, 141, 170, 208, 199, 74, 93, 166, 35, 81, 212, 86, 158, 162, 22, 175, 168, 213, 148, 254, 131, 79, 212, 162, 214, 255, 45, 231, 63, 104, 146, 47, 51, 18, 187, 156, 37, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 156, 211, 165, 107, 137, 142, 107, 233, 193, 15, 4, 182, 14, 206, 111, 124, 197, 24, 131, 13, 156, 218, 0, 253, 11, 83, 182, 227, 29, 64, 114, 33, 130, 86, 230, 58, 170, 150, 224, 131, 197, 120, 88, 8, 23, 135, 212, 114, 126, 131, 34, 46, 40, 21, 179, 71, 152, 82, 176, 205, 106, 13, 229, 19, 48, 70, 6, 3, 85, 29, 17, 4, 63, 48, 61, 130, 59, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 104, 50, 112, 54, 114, 55, 107, 57, 101, 52, 106, 56, 97, 52, 112, 46, 114, 102, 56, 119, 46, 97, 117, 115, 116, 114, 97, 108, 105, 97, 101, 97, 115, 116, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 105, 111, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 111, 129, 198, 35, 161, 213, 26, 120, 1, 118, 171, 24, 218, 108, 27, 212, 193, 115, 224, 125, 114, 122, 57, 94, 2, 5, 206, 118, 98, 94, 138, 212, 7...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: Unexpected error:
<*fmt.wrapError | 0xc000e5a200>:
route host was never found: Get "https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
{
msg: "route host was never found: Get \"https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
err: <*url.Error | 0xc00111a8a0>{
Op: "Get",
URL: "https://agnhost-e2e-serving-app-r4zh5.apps.aro.h2p6r7k9e4j8a4p.rf8w.australiaeast.aroapp-hcp.io",
Err: <*tls.CertificateVerificationError | 0xc00111a870>{
UnverifiedCertificates: [
{
Raw: [48, 130, 3, 251, 48, 130, 2, 227, 160, 3, 2, 1, 2, 2, 8, 92, 203, 189, 73, 220, 104, 43, 31, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 48, 38, 49, 18, 48, 16, 6, 3, 85, 4, 11, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 16, 48, 14, 6, 3, 85, 4, 3, 19, 7, 114, 111, 111, 116, 45, 99, 97, 48, 30, 23, 13, 50, 54, 48, 51, 50, 54, 48, 54, 52, 53, 49, 48, 90, 23, 13, 50, 55, 48, 51, 50, 54, 48, 54, 52, 53, 49, 48, 90, 48, 48, 49, 18, 48, 16, 6, 3, 85, 4, 10, 19, 9, 111, 112, 101, 110, 115, 104, 105, 102, 116, 49, 26, 48, 24, 6, 3, 85, 4, 3, 19, 17, 111, 112, 101, 110, 115, 104, 105, 102, 116, 45, 105, 110, 103, 114, 101, 115, 115, 48, 130, 1, 34, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 1, 5, 0, 3, 130, 1, 15, 0, 48, 130, 1, 10, 2, 130, 1, 1, 0, 184, 249, 228, 29, 76, 100, 40, 200, 92, 162, 72, 142, 129, 115, 47, 12, 145, 183, 72, 240, 1, 5, 92, 53, 47, 31, 202, 10, 174, 62, 48, 18, 31, 123, 117, 140, 213, 164, 154, 238, 253, 232, 211, 79, 197, 28, 90, 200, 94, 224, 0, 70, 240, 119, 180, 27, 69, 87, 250, 57, 250, 182, 205, 211, 32, 26, 243, 182, 5, 144, 142, 58, 253, 89, 73, 124, 131, 53, 242, 32, 215, 113, 115, 161, 138, 70, 204, 131, 156, 64, 119, 133, 189, 247, 73, 99, 55, 100, 228, 240, 153, 178, 98, 32, 168, 45, 26, 44, 226, 43, 23, 138, 243, 25, 89, 222, 29, 108, 66, 218, 157, 9, 246, 107, 237, 142, 13, 120, 2, 51, 162, 36, 40, 113, 174, 6, 24, 142, 141, 102, 87, 120, 195, 146, 75, 183, 146, 189, 224, 113, 188, 226, 122, 101, 188, 79, 120, 110, 96, 64, 136, 125, 35, 111, 198, 186, 195, 167, 74, 193, 60, 173, 53, 229, 146, 21, 60, 205, 130, 8, 88, 103, 27, 128, 125, 81, 34, 171, 43, 156, 241, 130, 164, 77, 16, 73, 90, 75, 165, 203, 168, 125, 253, 236, 167, 195, 124, 242, 13, 239, 113, 109, 161, 225, 227, 92, 142, 4, 22, 72, 42, 37, 19, 217, 72, 151, 205, 233, 36, 126, 107, 7, 87, 232, 186, 158, 130, 205, 56, 29, 44, 197, 167, 114, 125, 3, 48, 242, 51, 4, 177, 248, 136, 183, 247, 237, 2, 3, 1, 0, 1, 163, 130, 1, 33, 48, 130, 1, 29, 48, 14, 6, 3, 85, 29, 15, 1, 1, 255, 4, 4, 3, 2, 5, 160, 48, 29, 6, 3, 85, 29, 37, 4, 22, 48, 20, 6, 8, 43, 6, 1, 5, 5, 7, 3, 2, 6, 8, 43, 6, 1, 5, 5, 7, 3, 1, 48, 12, 6, 3, 85, 29, 19, 1, 1, 255, 4, 2, 48, 0, 48, 73, 6, 3, 85, 29, 14, 4, 66, 4, 64, 12, 97, 255, 11, 226, 234, 126, 212, 87, 16, 110, 229, 159, 103, 200, 181, 174, 217, 7, 108, 202, 95, 92, 201, 13, 142, 12, 237, 141, 170, 208, 199, 74, 93, 166, 35, 81, 212, 86, 158, 162, 22, 175, 168, 213, 148, 254, 131, 79, 212, 162, 214, 255, 45, 231, 63, 104, 146, 47, 51, 18, 187, 156, 37, 48, 75, 6, 3, 85, 29, 35, 4, 68, 48, 66, 128, 64, 156, 211, 165, 107, 137, 142, 107, 233, 193, 15, 4, 182, 14, 206, 111, 124, 197, 24, 131, 13, 156, 218, 0, 253, 11, 83, 182, 227, 29, 64, 114, 33, 130, 86, 230, 58, 170, 150, 224, 131, 197, 120, 88, 8, 23, 135, 212, 114, 126, 131, 34, 46, 40, 21, 179, 71, 152, 82, 176, 205, 106, 13, 229, 19, 48, 70, 6, 3, 85, 29, 17, 4, 63, 48, 61, 130, 59, 42, 46, 97, 112, 112, 115, 46, 97, 114, 111, 46, 104, 50, 112, 54, 114, 55, 107, 57, 101, 52, 106, 56, 97, 52, 112, 46, 114, 102, 56, 119, 46, 97, 117, 115, 116, 114, 97, 108, 105, 97, 101, 97, 115, 116, 46, 97, 114, 111, 97, 112, 112, 45, 104, 99, 112, 46, 105, 111, 48, 13, 6, 9, 42, 134, 72, 134, 247, 13, 1, 1, 11, 5, 0, 3, 130, 1, 1, 0, 111, 129, 198, 35, 161, 213, 26, 120, 1, 118, 171, 24, 218, 108, 27, 212, 193, 115, 224, 125, 114, 122, 57, 94, 2, 5, 206, 118, 98, 94, 138, 212, 7...
Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.
Learn more here: https://onsi.github.io/gomega/#adjusting-output
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||