CIHealth
<
Single Day: Jun 19▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
Get "<url>": net/http: TLS handshake timeout
23:00:26 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:162]: failed to list nodes after AZ nodepool creation
Unexpected error:
    <*url.Error | 0xc0016e2240>: 
    Get "https://api.f2e6d6p6j1q8o4d.91nt.uksouth.aroapp-hcp.io:443/api/v1/nodes": net/http: TLS handshake timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:162]: failed to list nodes after AZ nodepool creation
Unexpected error:
    <*url.Error | 0xc0016e2240>: 
    Get "https://api.f2e6d6p6j1q8o4d.91nt.uksouth.aroapp-hcp.io:443/api/v1/nodes": net/http: TLS handshake timeout
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 12
e2e (12)
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
21:23:09 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-bm6qd9"
Unexpected error:
    <*fmt.wrapError | 0xc000702420>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-76lgc.apps.aro.n6x4q8i7r6l8a4d.ldfc.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-76lgc.apps.aro.n6x4q8i7r6l8a4d.ldfc.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-bm6qd9"
Unexpected error:
    <*fmt.wrapError | 0xc000702420>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-76lgc.apps.aro.n6x4q8i7r6l8a4d.ldfc.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-76lgc.apps.aro.n6x4q8i7r6l8a4d.ldfc.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
21:23:09 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-vc5p5t" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00160ec60>: 
    failed to create HCP cluster cluster-zstream-4-22-vc5p5t, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-vc5p5t in resource group rg-zstream-upgrade-4-22-xzbf97, error: failed waiting for cluster="cluster-zstream-4-22-vc5p5t" in resourcegroup="rg-zstream-upgrade-4-22-xzbf97" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-vc5p5t in resource group rg-zstream-upgrade-4-22-xzbf97, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-vc5p5t" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00160ec60>: 
    failed to create HCP cluster cluster-zstream-4-22-vc5p5t, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-vc5p5t in resource group rg-zstream-upgrade-4-22-xzbf97, error: failed waiting for cluster="cluster-zstream-4-22-vc5p5t" in resourcegroup="rg-zstream-upgrade-4-22-xzbf97" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-vc5p5t in resource group rg-zstream-upgrade-4-22-xzbf97, error: context deadline exceeded
    ...
occurred
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
21:23:09 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-kmdgjv"
Unexpected error:
    <*fmt.wrapError | 0xc0009ea4c0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-pvsjg.apps.aro.z1b7r2v4p5j6g6y.dxfv.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-pvsjg.apps.aro.z1b7r2v4p5j6g6y.dxfv.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-kmdgjv"
Unexpected error:
    <*fmt.wrapError | 0xc0009ea4c0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-pvsjg.apps.aro.z1b7r2v4p5j6g6y.dxfv.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-pvsjg.apps.aro.z1b7r2v4p5j6g6y.dxfv.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
VerifySimpleWebApp failed: route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
21:23:09 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001900270>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-kt5nq.apps.aro.cilium-cl.i3eu.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-kt5nq.apps.aro.cilium-cl.i3eu.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001900270>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-kt5nq.apps.aro.cilium-cl.i3eu.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-kt5nq.apps.aro.cilium-cl.i3eu.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
VerifySimpleWebApp failed: route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
21:23:09 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001c2a060>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-gpqm7.apps.aro.cilium-cluster.95la.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-gpqm7.apps.aro.cilium-cluster.95la.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001c2a060>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-gpqm7.apps.aro.cilium-cluster.95la.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-gpqm7.apps.aro.cilium-cluster.95la.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
21:23:09 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-nlr5hm"
Unexpected error:
    <*fmt.wrapError | 0xc000f0a2a0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-2hbpj.apps.aro.m6l1q1f2b5a8o3l.la4x.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-2hbpj.apps.aro.m6l1q1f2b5a8o3l.la4x.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-nlr5hm"
Unexpected error:
    <*fmt.wrapError | 0xc000f0a2a0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-2hbpj.apps.aro.m6l1q1f2b5a8o3l.la4x.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-2hbpj.apps.aro.m6l1q1f2b5a8o3l.la4x.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True
21:23:09 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc00101e000>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc00101e000>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
Interrupted by User
21:23:09 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
Post "<url>": dial tcp <ip>:<port>: i/o timeout
21:23:09 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:236]: should be able to create SelfSubjectReview for admin credential 1
Unexpected error:
    <*url.Error | 0xc00130c030>: 
    Post "https://api.u6k7t4d6d9b4a5c.vf44.brazilsouth.aroapp-hcp.io:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:236]: should be able to create SelfSubjectReview for admin credential 1
Unexpected error:
    <*url.Error | 0xc00130c030>: 
    Post "https://api.u6k7t4d6d9b4a5c.vf44.brazilsouth.aroapp-hcp.io:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
21:23:09 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-cctxbf"
Unexpected error:
    <*fmt.wrapError | 0xc0005968c0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-fzhs2.apps.aro.j8y3s0f4z3i7b2z.g3rx.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-fzhs2.apps.aro.j8y3s0f4z3i7b2z.g3rx.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-cctxbf"
Unexpected error:
    <*fmt.wrapError | 0xc0005968c0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-fzhs2.apps.aro.j8y3s0f4z3i7b2z.g3rx.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-fzhs2.apps.aro.j8y3s0f4z3i7b2z.g3rx.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
dial tcp: lookup console-openshift-console
21:23:09 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.006s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc000bcba90>: 
    dial tcp: lookup console-openshift-console.apps.aro.b0n9v3c0r2b5h8v.2xda.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.006s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc000bcba90>: 
    dial tcp: lookup console-openshift-console.apps.aro.b0n9v3c0r2b5h8v.2xda.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
21:23:09 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:191]: failed to create node pool version 4.19.7 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc001776480>: 
    failed waiting for nodepool="np-ver-4-19-7" for cluster "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-9r69jz" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:191]: failed to create node pool version 4.19.7 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc001776480>: 
    failed waiting for nodepool="np-ver-4-19-7" for cluster "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-9r69jz" to finish creating: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
tls: failed to verify certificate: x509: certificate signed by unknown authority
20:10:01 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-4rkx7j"
Unexpected error:
    <*fmt.wrapError | 0xc000e0a620>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-9rwln.apps.aro.s0y5l7a1j6m3k3z.r0ib.australiaeast.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-4rkx7j"
Unexpected error:
    <*fmt.wrapError | 0xc000e0a620>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-9rwln.apps.aro.s0y5l7a1j6m3k3z.r0ib.australiaeast.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
20:10:01 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-qkdlc9" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00157e5a0>: 
    failed to create HCP cluster cluster-zstream-4-22-qkdlc9, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-qkdlc9 in resource group rg-zstream-upgrade-4-22-2vwndt, error: failed waiting for cluster="cluster-zstream-4-22-qkdlc9" in resourcegroup="rg-zstream-upgrade-4-22-2vwndt" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-qkdlc9 in resource group rg-zstream-upgrade-4-22-2vwndt, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-qkdlc9" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00157e5a0>: 
    failed to create HCP cluster cluster-zstream-4-22-qkdlc9, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-qkdlc9 in resource group rg-zstream-upgrade-4-22-2vwndt, error: failed waiting for cluster="cluster-zstream-4-22-qkdlc9" in resourcegroup="rg-zstream-upgrade-4-22-2vwndt" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-qkdlc9 in resource group rg-zstream-upgrade-4-22-2vwndt, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 8
e2e (8)
Cluster should become ready within '20.000000' minutes
20:07:04 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.005s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.005s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
timeout during CreateHCPClusterAndWait; context deadline exceeded
20:07:04 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapErrors | 0xc000bf22a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-b569b7" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-b569b7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapErrors | 0xc000bf22a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-b569b7" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-b569b7, error: context deadline exceeded
    ...
occurred
cluster has not yet reached Succeeded state
20:07:04 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
timeout during CreateHCPClusterAndWait; context deadline exceeded
20:07:04 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapErrors | 0xc0013b0930>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-6t4bbg, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-6t4bbg" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-6t4bbg, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapErrors | 0xc0013b0930>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-6t4bbg, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-6t4bbg" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-6t4bbg, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
20:07:04 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapErrors | 0xc0015ced50>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ccr9nc, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-ccr9nc" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ccr9nc, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapErrors | 0xc0015ced50>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ccr9nc, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-ccr9nc" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-ccr9nc, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
20:07:04 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
    <*fmt.wrapErrors | 0xc0010a0cc0>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-cgxj45" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-cgxj45, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
    <*fmt.wrapErrors | 0xc0010a0cc0>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-cgxj45" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-cgxj45, error: context deadline exceeded
    ...
occurred
failed to wait for first cluster "<cluster>" to complete creation (timeout '20.000000' minutes)
20:07:04 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
20:07:04 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapErrors | 0xc0011128a0>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-m5mrms, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-m5mrms" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-m5mrms, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapErrors | 0xc0011128a0>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-m5mrms, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-m5mrms" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-m5mrms, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 7
e2e (7)
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
16:14:56 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: failed to verify simple web app on back-level cluster version 4.19
Unexpected error:
    <*fmt.wrapError | 0xc000ee0200>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-6wbqg.apps.aro.a0r2x9t6a1n8h3a.vojx.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-6wbqg.apps.aro.a0r2x9t6a1n8h3a.vojx.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: failed to verify simple web app on back-level cluster version 4.19
Unexpected error:
    <*fmt.wrapError | 0xc000ee0200>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-6wbqg.apps.aro.a0r2x9t6a1n8h3a.vojx.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-6wbqg.apps.aro.a0r2x9t6a1n8h3a.vojx.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
16:14:56 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-6gwhvw/cluster-candidate-5-0-t9nxss should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0010e8a80>: 
    failed to create HCP cluster cluster-candidate-5-0-t9nxss, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-t9nxss in resource group rg-candidate-5-0-6gwhvw, error: failed waiting for cluster="cluster-candidate-5-0-t9nxss" in resourcegroup="rg-candidate-5-0-6gwhvw" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-t9nxss in resource group rg-candidate-5-0-6gwhvw, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-6gwhvw/cluster-candidate-5-0-t9nxss should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0010e8a80>: 
    failed to create HCP cluster cluster-candidate-5-0-t9nxss, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-t9nxss in resource group rg-candidate-5-0-6gwhvw, error: failed waiting for cluster="cluster-candidate-5-0-t9nxss" in resourcegroup="rg-candidate-5-0-6gwhvw" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-t9nxss in resource group rg-candidate-5-0-6gwhvw, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message Get "<url>": net/http: TLS handshake timeout; provider Microsoft.Compute
16:14:56 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:168]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc001910860>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-td4tgf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "name": "56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "status": "Failed",
      "startTime": "2026-06-19T18:20:56.860797749Z",
      "endTime": "2026-06-19T18:22:33.081221493Z",
      "error": {
        "code": "InternalServerError",
        "message": "Get \"https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus?%24filter=location+eq+%27brazilsouth%27\u0026api-version=2021-07-01\": net/http: TLS handshake timeout"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:168]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc001910860>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-td4tgf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "name": "56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "status": "Failed",
      "startTime": "2026-06-19T18:20:56.860797749Z",
      "endTime": "2026-06-19T18:22:33.081221493Z",
      "error": {
        "code": "InternalServerError",
        "message": "Get \"https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus?%24filter=location+eq+%27brazilsouth%27\u0026api-version=2021-07-01\": net/http: TLS handshake timeout"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
16:14:56 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-xcrwzf"
Unexpected error:
    <*fmt.wrapError | 0xc000f02e60>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-nkmh6.apps.aro.e0z4x4d2y5v6c4w.ci57.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-nkmh6.apps.aro.e0z4x4d2y5v6c4w.ci57.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-xcrwzf"
Unexpected error:
    <*fmt.wrapError | 0xc000f02e60>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-nkmh6.apps.aro.e0z4x4d2y5v6c4w.ci57.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-nkmh6.apps.aro.e0z4x4d2y5v6c4w.ci57.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
Deserialization Error: no output from command
16:14:56 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
tls.(*Conn).readRecordOrCCS(0xc000fec388, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc001701c60 sp=0xc0017019e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000fec388, {0xc0013d4000, 0x1000, 0xc000f0a6c0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc001701cc8 sp=0xc001701c60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc0012d1200, {0xc0013d4000?, 0x32dcb05?, 0x9219480?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc001701d28 sp=0xc001701cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc000f08780)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc001701d60 sp=0xc001701d28 pc=0x3084943
bufio.(*Reader).Peek(0xc000f08780, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc001701d80 sp=0xc001701d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0012d1200)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc001701fc8 sp=0xc001701d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc001701fe0 sp=0xc001701fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001701fe8 sp=0xc001701fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 227
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 236 gp=0xc001102e00 m=nil [select]:
runtime.gopark(0xc0013adf48?, 0x2?, 0xc0?, 0xdd?, 0xc0013adef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0013add78 sp=0xc0013add58 pc=0x2f1e50e
runtime.selectgo(0xc0013adf48, 0xc0013adef0, 0xc000f96bc0?, 0x0, 0x0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc0013adeb8 sp=0xc0013add78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc0012d1200)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc0013adfc8 sp=0xc0013adeb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc0013adfe0 sp=0xc0013adfc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0013adfe8 sp=0xc0013adfe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 227
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 193 gp=0xc00064ba40 m=nil [sync.WaitGroup.Wait]:
runtime.gopark(0xc00149ba60?, 0x2eb5874?, 0x0?, 0xa3?, 0x7fe57c39bb80?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc00149ba00 sp=0xc00149b9e0 pc=0x2f1e50e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc000f974d0, 0x0, 0x1, 0x0, 0x19)
	/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc00149ba68 sp=0xc00149ba00 pc=0x2efa59d
sync.runtime_SemacquireWaitGroup(0xc000013d88?, 0x40?)
	/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc00149baa0 sp=0xc00149ba68 pc=0x2f1fc2e
sync.(*WaitGroup).Wait(0xc000f974c8)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc00149bac8 sp=0xc00149baa0 pc=0x2f43405
golang.org/x/sync/errgroup.(*Group).Wait(0xc000f974c0)
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc00149bae8 sp=0xc00149bac8 pc=0x4a4bcde
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610({0x7fe57d449c60, 0xc000652600}, 0xc000bf6d38, {0xc00092fab8, 0x12}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:525 +0x8de fp=0xc00149bd90 sp=0xc00149bae8 pc=0x6c31fbe
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000e56000, {0x7fe57d449c60, 0xc000652600}, {0xc00092fab8, 0x12}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:539 +0x29d fp=0xc00149bf08 sp=0xc00149bd90 pc=0x6c549fd
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7fe57d449c60?, 0xc000652600?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:362 +0x151 fp=0xc00149bfc0 sp=0xc00149bf08 pc=0x6c52a91
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x28 fp=0xc00149bfe0 sp=0xc00149bfc0 pc=0x6c52908
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00149bfe8 sp=0xc00149bfe0 pc=0x2f26cc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 226
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:355 +0x98

rax    0xfffffffffffffffc
rbx    0x5
rcx    0x2e9de2e
rdx    0x80
rdi    0x5
rsi    0x7fff500886d4
rbp    0x7fff50088688
rsp    0x7fff50088640
r8     0x0
r9     0x0
r10    0x1cb3
r11    0x246
r12    0x7fff50088718
r13    0x1
r14    0xc9086e0
r15    0x3
rip    0x2e9de2e
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
tls.(*Conn).readRecordOrCCS(0xc000fec388, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc001701c60 sp=0xc0017019e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000fec388, {0xc0013d4000, 0x1000, 0xc000f0a6c0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc001701cc8 sp=0xc001701c60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc0012d1200, {0xc0013d4000?, 0x32dcb05?, 0x9219480?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc001701d28 sp=0xc001701cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc000f08780)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc001701d60 sp=0xc001701d28 pc=0x3084943
bufio.(*Reader).Peek(0xc000f08780, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc001701d80 sp=0xc001701d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0012d1200)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc001701fc8 sp=0xc001701d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc001701fe0 sp=0xc001701fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001701fe8 sp=0xc001701fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 227
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 236 gp=0xc001102e00 m=nil [select]:
runtime.gopark(0xc0013adf48?, 0x2?, 0xc0?, 0xdd?, 0xc0013adef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0013add78 sp=0xc0013add58 pc=0x2f1e50e
runtime.selectgo(0xc0013adf48, 0xc0013adef0, 0xc000f96bc0?, 0x0, 0x0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc0013adeb8 sp=0xc0013add78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc0012d1200)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc0013adfc8 sp=0xc0013adeb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc0013adfe0 sp=0xc0013adfc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0013adfe8 sp=0xc0013adfe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 227
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 193 gp=0xc00064ba40 m=nil [sync.WaitGroup.Wait]:
runtime.gopark(0xc00149ba60?, 0x2eb5874?, 0x0?, 0xa3?, 0x7fe57c39bb80?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc00149ba00 sp=0xc00149b9e0 pc=0x2f1e50e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc000f974d0, 0x0, 0x1, 0x0, 0x19)
	/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc00149ba68 sp=0xc00149ba00 pc=0x2efa59d
sync.runtime_SemacquireWaitGroup(0xc000013d88?, 0x40?)
	/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc00149baa0 sp=0xc00149ba68 pc=0x2f1fc2e
sync.(*WaitGroup).Wait(0xc000f974c8)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc00149bac8 sp=0xc00149baa0 pc=0x2f43405
golang.org/x/sync/errgroup.(*Group).Wait(0xc000f974c0)
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc00149bae8 sp=0xc00149bac8 pc=0x4a4bcde
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610({0x7fe57d449c60, 0xc000652600}, 0xc000bf6d38, {0xc00092fab8, 0x12}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:525 +0x8de fp=0xc00149bd90 sp=0xc00149bae8 pc=0x6c31fbe
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000e56000, {0x7fe57d449c60, 0xc000652600}, {0xc00092fab8, 0x12}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:539 +0x29d fp=0xc00149bf08 sp=0xc00149bd90 pc=0x6c549fd
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7fe57d449c60?, 0xc000652600?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:362 +0x151 fp=0xc00149bfc0 sp=0xc00149bf08 pc=0x6c52a91
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x28 fp=0xc00149bfe0 sp=0xc00149bfc0 pc=0x6c52908
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00149bfe8 sp=0xc00149bfe0 pc=0x2f26cc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 226
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:355 +0x98

rax    0xfffffffffffffffc
rbx    0x5
rcx    0x2e9de2e
rdx    0x80
rdi    0x5
rsi    0x7fff500886d4
rbp    0x7fff50088688
rsp    0x7fff50088640
r8     0x0
r9     0x0
r10    0x1cb3
r11    0x246
r12    0x7fff50088718
r13    0x1
r14    0xc9086e0
r15    0x3
rip    0x2e9de2e
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
16:14:56 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:126]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000be83c0>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-l9sf6s, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-l9sf6s" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:126]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000be83c0>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-l9sf6s, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-l9sf6s" to finish creating: context deadline exceeded
    ...
occurred
Interrupted by User
16:14:56 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 12
e2e (12)
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
12:37:07 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: failed to verify simple web app on back-level cluster version 4.19
Unexpected error:
    <*fmt.wrapError | 0xc0000d4e80>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-zdzw6.apps.aro.w6e9g9x3b8d2a6n.cvgp.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-zdzw6.apps.aro.w6e9g9x3b8d2a6n.cvgp.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: failed to verify simple web app on back-level cluster version 4.19
Unexpected error:
    <*fmt.wrapError | 0xc0000d4e80>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-zdzw6.apps.aro.w6e9g9x3b8d2a6n.cvgp.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-zdzw6.apps.aro.w6e9g9x3b8d2a6n.cvgp.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
12:37:07 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:138]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc000948600>: 
    failed to create NodePool np-2, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-2 in resource group cx-rg-cluster-x5k42b, error: failed waiting for nodepool="np-2" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-x5k42b" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:138]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc000948600>: 
    failed to create NodePool np-2, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-2 in resource group cx-rg-cluster-x5k42b, error: failed waiting for nodepool="np-2" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-x5k42b" to finish creating: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
12:37:07 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:111]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapErrors | 0xc0012f6870>: 
    failed to create HCP cluster ephemeral-disk, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-pmrxbp, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-pmrxbp" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-pmrxbp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:111]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapErrors | 0xc0012f6870>: 
    failed to create HCP cluster ephemeral-disk, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-pmrxbp, error: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-pmrxbp" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ephemeral-disk in resource group ephemeral-osdisk-pmrxbp, error: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
12:37:07 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:484]: failed to create nodepool nps-420-422 at version 4.20.26
Unexpected error:
    <*fmt.wrapErrors | 0xc000980b40>: 
    failed to create NodePool nps-420-422, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nps-420-422 in resource group rg-np-skip-minor-b69dlq-gjbzkc, error: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-b69dlq" in resourcegroup="rg-np-skip-minor-b69dlq-gjbzkc" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:484]: failed to create nodepool nps-420-422 at version 4.20.26
Unexpected error:
    <*fmt.wrapErrors | 0xc000980b40>: 
    failed to create NodePool nps-420-422, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nps-420-422 in resource group rg-np-skip-minor-b69dlq-gjbzkc, error: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-b69dlq" in resourcegroup="rg-np-skip-minor-b69dlq-gjbzkc" to finish creating: context deadline exceeded
    ...
occurred
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
12:37:07 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-d5mvfb"
Unexpected error:
    <*fmt.wrapError | 0xc000868780>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-c5mpv.apps.aro.q6k1z8g8f4f4u6a.jwx7.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-c5mpv.apps.aro.q6k1z8g8f4f4u6a.jwx7.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-d5mvfb"
Unexpected error:
    <*fmt.wrapError | 0xc000868780>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-c5mpv.apps.aro.q6k1z8g8f4f4u6a.jwx7.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-c5mpv.apps.aro.q6k1z8g8f4f4u6a.jwx7.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
client rate limiter Wait returned an error: context deadline exceeded
12:37:07 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc000c57710>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc000c57710>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
12:37:07 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-w9rgfw"
Unexpected error:
    <*fmt.wrapError | 0xc0011627c0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-kl6kj.apps.aro.j2f9t3b7s3i2i2w.6dbg.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-kl6kj.apps.aro.j2f9t3b7s3i2i2w.6dbg.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-w9rgfw"
Unexpected error:
    <*fmt.wrapError | 0xc0011627c0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-kl6kj.apps.aro.j2f9t3b7s3i2i2w.6dbg.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-kl6kj.apps.aro.j2f9t3b7s3i2i2w.6dbg.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
Get "<url>": dial tcp <ip>:<port>: i/o timeout
12:37:07 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to get ClusterVersion resource
Unexpected error:
    <*url.Error | 0xc000b63410>: 
    Get "https://api.i2o1x2r3h8b1j4g.mbnd.brazilsouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to get ClusterVersion resource
Unexpected error:
    <*url.Error | 0xc000b63410>: 
    Get "https://api.i2o1x2r3h8b1j4g.mbnd.brazilsouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
12:37:07 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-6x7bgt/cluster-candidate-5-0-9xqv7k should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0005b8240>: 
    failed to create HCP cluster cluster-candidate-5-0-9xqv7k, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-9xqv7k in resource group rg-candidate-5-0-6x7bgt, error: failed waiting for cluster="cluster-candidate-5-0-9xqv7k" in resourcegroup="rg-candidate-5-0-6x7bgt" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-9xqv7k in resource group rg-candidate-5-0-6x7bgt, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-6x7bgt/cluster-candidate-5-0-9xqv7k should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0005b8240>: 
    failed to create HCP cluster cluster-candidate-5-0-9xqv7k, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-9xqv7k in resource group rg-candidate-5-0-6x7bgt, error: failed waiting for cluster="cluster-candidate-5-0-9xqv7k" in resourcegroup="rg-candidate-5-0-6x7bgt" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-9xqv7k in resource group rg-candidate-5-0-6x7bgt, error: context deadline exceeded
    ...
occurred
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
12:37:07 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-zt2bhx"
Unexpected error:
    <*fmt.wrapError | 0xc00045e280>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-lnfss.apps.aro.t1r3e7r0v3t4w2w.9gid.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-lnfss.apps.aro.t1r3e7r0v3t4w2w.9gid.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-zt2bhx"
Unexpected error:
    <*fmt.wrapError | 0xc00045e280>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-lnfss.apps.aro.t1r3e7r0v3t4w2w.9gid.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-lnfss.apps.aro.t1r3e7r0v3t4w2w.9gid.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
12:37:07 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:406 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:406 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: ...
to be empty
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
12:37:07 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:138]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc001027260>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-cnkjd2" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:138]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc001027260>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-cnkjd2" to finish creating: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 24
e2e (24)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
11:16:27 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00108a3f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-5vzdx7" in resourcegroup="rg-np-dg-minor-5vzdx7-xzzbj6" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00108a3f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-5vzdx7" in resourcegroup="rg-np-dg-minor-5vzdx7-xzzbj6" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:89]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc0017106c0>: 
    failed to create HCP cluster cx-rg-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-gzhwj2, error: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-gzhwj2" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-gzhwj2, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:89]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc0017106c0>: 
    failed to create HCP cluster cx-rg-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-gzhwj2, error: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-gzhwj2" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cx-rg-hcp-cluster in resource group cx-rg-cluster-gzhwj2, error: context deadline exceeded
    ...
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
11:16:27 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0017384c8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-gt9g9n" in resourcegroup="rg-np-version-upgrade-gt9g9n-2cbnk4" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0017384c8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-gt9g9n" in resourcegroup="rg-np-version-upgrade-gt9g9n-2cbnk4" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
11:16:27 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:137]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc0018061e0>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-7t4tb6, error: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-7t4tb6" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:137]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc0018061e0>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group tls-endpoint-cluster-7t4tb6, error: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-7t4tb6" to finish creating: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:79]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapErrors | 0xc001406060>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zhbj8b, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-zhbj8b" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zhbj8b, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:79]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapErrors | 0xc001406060>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zhbj8b, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-zhbj8b" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zhbj8b, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:95]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000773500>: 
    failed to create HCP cluster np-autoscale-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-gn2pq5, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-gn2pq5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-gn2pq5, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:95]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000773500>: 
    failed to create HCP cluster np-autoscale-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-gn2pq5, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-gn2pq5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-gn2pq5, error: context deadline exceeded
    ...
occurred
Cluster should become ready within '20.000000' minutes
11:16:27 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.005s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.005s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
Interrupted by User
11:16:27 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
failed to wait for first cluster "<cluster>" to complete creation (timeout '20.000000' minutes)
11:16:27 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:149]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapErrors | 0xc0017a7dd0>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-bg8h4r, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bg8h4r" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-bg8h4r, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:149]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapErrors | 0xc0017a7dd0>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-bg8h4r, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bg8h4r" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-bg8h4r, error: context deadline exceeded
    ...
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
11:16:27 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0013fc318>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-9nrqzg" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0013fc318>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-9nrqzg" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
11:16:27 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0018ba438>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-5wpd8t" in resourcegroup="rg-np-skip-minor-5wpd8t-4p6jxn" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0018ba438>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-5wpd8t" in resourcegroup="rg-np-skip-minor-5wpd8t-4p6jxn" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapErrors | 0xc000f5a090>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-gzbbft, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-gzbbft" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-gzbbft, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:112]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapErrors | 0xc000f5a090>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-gzbbft, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-gzbbft" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-gzbbft, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapErrors | 0xc0013398f0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-zlrhrz" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zlrhrz, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapErrors | 0xc0013398f0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-zlrhrz" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zlrhrz, error: context deadline exceeded
    ...
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
11:16:27 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001632678>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-k7ntz6" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001632678>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-k7ntz6" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc0009a8600>: 
    failed to create HCP cluster basic-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-qw2ps9, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-qw2ps9" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-qw2ps9, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc0009a8600>: 
    failed to create HCP cluster basic-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-qw2ps9, error: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-qw2ps9" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-cluster in resource group rg-cluster-nsg-subnet-reuse-qw2ps9, error: context deadline exceeded
    ...
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
11:16:27 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012fe150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-6b2hnp" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012fe150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-6b2hnp" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapErrors | 0xc001660240>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-qxn4hc" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-qxn4hc, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapErrors | 0xc001660240>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-qxn4hc" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster cilium-cluster in resource group complex-cilium-kv-qxn4hc, error: context deadline exceeded
    ...
occurred
cluster has not yet reached Succeeded state
11:16:27 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.008s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.008s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapErrors | 0xc0015d0a80>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-frz9v7, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-frz9v7" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-frz9v7, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapErrors | 0xc0015d0a80>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-frz9v7, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-frz9v7" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-frz9v7, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:89]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc00141e6c0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-vkdp8j, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-vkdp8j" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-vkdp8j, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:89]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc00141e6c0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-vkdp8j, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-vkdp8j" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-vkdp8j, error: context deadline exceeded
    ...
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
11:16:27 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001130198>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-hcwf6w" in resourcegroup="rg-np-version-upgrade-hcwf6w-flzhzz" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001130198>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-hcwf6w" in resourcegroup="rg-np-version-upgrade-hcwf6w-flzhzz" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
11:16:27 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapErrors | 0xc0013cc510>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-vdvb7c, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vdvb7c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-vdvb7c, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapErrors | 0xc0013cc510>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-vdvb7c, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-vdvb7c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-vdvb7c, error: context deadline exceeded
    ...
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
11:16:27 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00105e5e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-bfr57d" in resourcegroup="rg-np-version-upgrade-bfr57d-2zkjv8" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00105e5e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-bfr57d" in resourcegroup="rg-np-version-upgrade-bfr57d-2zkjv8" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
timeout during CreateHCPClusterAndWait; context deadline exceeded
09:12:23 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:596]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000b14c60>: 
    failed to create HCP cluster np-downgrade-29tqx9, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-downgrade-29tqx9 in resource group rg-np-downgrade-29tqx9-z5nf6j, error: failed waiting for cluster="np-downgrade-29tqx9" in resourcegroup="rg-np-downgrade-29tqx9-z5nf6j" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-downgrade-29tqx9 in resource group rg-np-downgrade-29tqx9-z5nf6j, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:596]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000b14c60>: 
    failed to create HCP cluster np-downgrade-29tqx9, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-downgrade-29tqx9 in resource group rg-np-downgrade-29tqx9-z5nf6j, error: failed waiting for cluster="np-downgrade-29tqx9" in resourcegroup="rg-np-downgrade-29tqx9-z5nf6j" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-downgrade-29tqx9 in resource group rg-np-downgrade-29tqx9-z5nf6j, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
08:38:44 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:259]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000729080>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-2trd5v, error: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-2trd5v" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:259]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000729080>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group e2e-cidr-connectivity-2trd5v, error: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-2trd5v" to finish creating: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
x509: certificate signed by unknown authority
04:04:33 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.003s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.003s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
node pool <nodepool> verification failed: <node> (release images unchanged: [quay
04:04:33 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:220]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.errorString | 0xc00081af90>: 
    node pool upgrade verification failed: e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk (release images unchanged: [quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258 quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f quay.io/openshift/origin-insights-runtime-extractor:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13 quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184 quay.io/openshift/origin-kube-rbac-proxy:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9 arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1 quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef quay.io/openshift/origin-insights-runtime-exporter:latest]); e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq (not ready); nodes=[{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk","ready":true,"containerRuntimeVersion":"cri-o://1.34.2-2.rhaos4.21.gitc8e8b46.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-4xsj8","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44c807272cb410a84d347dda34dd77990697384e450b7b832e533e2a2352de76","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48c6e2725865ef2b8b7d6dc501c1778d681ed1eaf8bccad0d978c163b36a8ad9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad2768f4eed20ecddef9652e72c2536a9c5a129a733ccb8c423b8421b1d047bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd7c7b127ef0c7067d52f4ae8f3f95f878ec1c8b78a64fbb89f184824211ed80","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1110f6e8d66ba68224f9a7eedf1baa54de2d0c58332f611862242fe8ff311295","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1459af715cdfb7e21e056d793478f4387d077ad3d4d4e095c250c10eef809631","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29dd11a9e6e7344676611823c362862070b2ebc080793514aa14178527bfa7f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f35aba18e4a81d2e301162430d451d147378a5bd74a711135bead02921256554","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e18b7e1a7e67c4cff5e27d62c7de71968108b4366f8e30619fe6d277288dd63","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b61249fb7f26b253b8e634ef8cc47aa599b108e74651dba2c0dbda94df82f30","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab245b1c592c3179b63e3a79fa641f8125c16a29a67338f9adb489fb85af87b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cd91d2f0e56a7e3484ad8718845f9747c94180b0ee975fb62ffc75df214577b","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16139e0510e8b95cab7c2645e63f08d069c15ed64553e5133b99e59736ead4df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84113d4695be785283aacfc42fb9905bb7cd9233c4157f7d09726e641ea0d648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:761ccd3ced18e3e3d3760ece8e13bc07eb6119fdc05e4e5f7677d2695ab99ff4","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d6207a89a90d9f3408da1ec88e8c13c8c23330faf4c82be14c1e07b9ddb44a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-9txmg","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00f6ba985494302f2d431ad27b9bc6300d0c0d2dd0433e6403525e427a3341fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5469f5ec20ac2e7a2d82e84767c1c635671fce4ae3d7c27402289254f1183e04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a127e2a436b84937d2d970f3c316d82f61da498f04ec6036a91ad4284403f913","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d69fdb4df0385545fea609e6665de6c4b6dd4763b73a35a9d49456f879987925","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84493322067d59517be692eb5b1ffc90ae87f1a382459de635a69db4520e2750","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0476037c9c3b76a0135bc565733743cda8777479dc92a5cf2e6b79e999d35e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq","ready":false,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:220]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.errorString | 0xc00081af90>: 
    node pool upgrade verification failed: e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk (release images unchanged: [quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258 quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f quay.io/openshift/origin-insights-runtime-extractor:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13 quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184 quay.io/openshift/origin-kube-rbac-proxy:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9 arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1 quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef quay.io/openshift/origin-insights-runtime-exporter:latest]); e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq (not ready); nodes=[{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk","ready":true,"containerRuntimeVersion":"cri-o://1.34.2-2.rhaos4.21.gitc8e8b46.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-4xsj8","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44c807272cb410a84d347dda34dd77990697384e450b7b832e533e2a2352de76","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48c6e2725865ef2b8b7d6dc501c1778d681ed1eaf8bccad0d978c163b36a8ad9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad2768f4eed20ecddef9652e72c2536a9c5a129a733ccb8c423b8421b1d047bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd7c7b127ef0c7067d52f4ae8f3f95f878ec1c8b78a64fbb89f184824211ed80","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1110f6e8d66ba68224f9a7eedf1baa54de2d0c58332f611862242fe8ff311295","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1459af715cdfb7e21e056d793478f4387d077ad3d4d4e095c250c10eef809631","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29dd11a9e6e7344676611823c362862070b2ebc080793514aa14178527bfa7f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f35aba18e4a81d2e301162430d451d147378a5bd74a711135bead02921256554","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e18b7e1a7e67c4cff5e27d62c7de71968108b4366f8e30619fe6d277288dd63","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b61249fb7f26b253b8e634ef8cc47aa599b108e74651dba2c0dbda94df82f30","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab245b1c592c3179b63e3a79fa641f8125c16a29a67338f9adb489fb85af87b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cd91d2f0e56a7e3484ad8718845f9747c94180b0ee975fb62ffc75df214577b","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16139e0510e8b95cab7c2645e63f08d069c15ed64553e5133b99e59736ead4df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84113d4695be785283aacfc42fb9905bb7cd9233c4157f7d09726e641ea0d648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:761ccd3ced18e3e3d3760ece8e13bc07eb6119fdc05e4e5f7677d2695ab99ff4","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d6207a89a90d9f3408da1ec88e8c13c8c23330faf4c82be14c1e07b9ddb44a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-9txmg","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00f6ba985494302f2d431ad27b9bc6300d0c0d2dd0433e6403525e427a3341fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5469f5ec20ac2e7a2d82e84767c1c635671fce4ae3d7c27402289254f1183e04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a127e2a436b84937d2d970f3c316d82f61da498f04ec6036a91ad4284403f913","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d69fdb4df0385545fea609e6665de6c4b6dd4763b73a35a9d49456f879987925","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84493322067d59517be692eb5b1ffc90ae87f1a382459de635a69db4520e2750","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0476037c9c3b76a0135bc565733743cda8777479dc92a5cf2e6b79e999d35e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq","ready":false,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678"]}]
    ...
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
01:21:52 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc001489e30>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-7bdkmf, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-7bdkmf" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc001489e30>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-7bdkmf, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-7bdkmf" to finish creating: context deadline exceeded
    ...
occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
01:21:52 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-fsrk9t" for cluster "cluster-candidate-4-20-fsrk9t"
Unexpected error:
    <*fmt.wrapErrors | 0xc00092e5a0>: 
    failed to create NodePool np-fsrk9t, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-fsrk9t in resource group rg-candidate-4-20-ph9zpk, error: failed waiting for nodepool="np-fsrk9t" for cluster "cluster-candidate-4-20-fsrk9t" in resourcegroup="rg-candidate-4-20-ph9zpk" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-fsrk9t" for cluster "cluster-candidate-4-20-fsrk9t"
Unexpected error:
    <*fmt.wrapErrors | 0xc00092e5a0>: 
    failed to create NodePool np-fsrk9t, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-fsrk9t in resource group rg-candidate-4-20-ph9zpk, error: failed waiting for nodepool="np-fsrk9t" for cluster "cluster-candidate-4-20-fsrk9t" in resourcegroup="rg-candidate-4-20-ph9zpk" to finish creating: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a