CIHealth
<
Single Day: Jun 19▾
Last 7 Days Weekly: Jun 15 - Jun 21 Sprint: Jun 8 - Jun 21 Single Day: Jun 19
Custom time window
> Reset
View JSON API

Environment: PROD

Failure patterns: 22 · Failures matched: 72 · Total runs: 21 · Runs affected: 11

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e14.76%1 of 21 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsJun 13: 2 · Jun 14: 0 · Jun 15: 1 · Jun 16: 2 · Jun 17: 0 · Jun 18: 1 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc000c57710>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc000c57710>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067949798622760960
Deserialization Error: no output from command
full failure pattern:
Deserialization Error: no output from command
Signal: NoiseSignal: Noise — source deserialization/no-output error — New failure pattern — no prior history
e2e14.76%1 of 21 job runs affectedNoiseSignal: Noise — source deserialization/no-output error — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
tls.(*Conn).readRecordOrCCS(0xc000fec388, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc001701c60 sp=0xc0017019e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000fec388, {0xc0013d4000, 0x1000, 0xc000f0a6c0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc001701cc8 sp=0xc001701c60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc0012d1200, {0xc0013d4000?, 0x32dcb05?, 0x9219480?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc001701d28 sp=0xc001701cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc000f08780)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc001701d60 sp=0xc001701d28 pc=0x3084943
bufio.(*Reader).Peek(0xc000f08780, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc001701d80 sp=0xc001701d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0012d1200)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc001701fc8 sp=0xc001701d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc001701fe0 sp=0xc001701fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001701fe8 sp=0xc001701fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 227
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 236 gp=0xc001102e00 m=nil [select]:
runtime.gopark(0xc0013adf48?, 0x2?, 0xc0?, 0xdd?, 0xc0013adef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0013add78 sp=0xc0013add58 pc=0x2f1e50e
runtime.selectgo(0xc0013adf48, 0xc0013adef0, 0xc000f96bc0?, 0x0, 0x0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc0013adeb8 sp=0xc0013add78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc0012d1200)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc0013adfc8 sp=0xc0013adeb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc0013adfe0 sp=0xc0013adfc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0013adfe8 sp=0xc0013adfe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 227
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 193 gp=0xc00064ba40 m=nil [sync.WaitGroup.Wait]:
runtime.gopark(0xc00149ba60?, 0x2eb5874?, 0x0?, 0xa3?, 0x7fe57c39bb80?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc00149ba00 sp=0xc00149b9e0 pc=0x2f1e50e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc000f974d0, 0x0, 0x1, 0x0, 0x19)
	/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc00149ba68 sp=0xc00149ba00 pc=0x2efa59d
sync.runtime_SemacquireWaitGroup(0xc000013d88?, 0x40?)
	/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc00149baa0 sp=0xc00149ba68 pc=0x2f1fc2e
sync.(*WaitGroup).Wait(0xc000f974c8)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc00149bac8 sp=0xc00149baa0 pc=0x2f43405
golang.org/x/sync/errgroup.(*Group).Wait(0xc000f974c0)
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc00149bae8 sp=0xc00149bac8 pc=0x4a4bcde
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610({0x7fe57d449c60, 0xc000652600}, 0xc000bf6d38, {0xc00092fab8, 0x12}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:525 +0x8de fp=0xc00149bd90 sp=0xc00149bae8 pc=0x6c31fbe
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000e56000, {0x7fe57d449c60, 0xc000652600}, {0xc00092fab8, 0x12}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:539 +0x29d fp=0xc00149bf08 sp=0xc00149bd90 pc=0x6c549fd
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7fe57d449c60?, 0xc000652600?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:362 +0x151 fp=0xc00149bfc0 sp=0xc00149bf08 pc=0x6c52a91
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x28 fp=0xc00149bfe0 sp=0xc00149bfc0 pc=0x6c52908
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00149bfe8 sp=0xc00149bfe0 pc=0x2f26cc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 226
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:355 +0x98

rax    0xfffffffffffffffc
rbx    0x5
rcx    0x2e9de2e
rdx    0x80
rdi    0x5
rsi    0x7fff500886d4
rbp    0x7fff50088688
rsp    0x7fff50088640
r8     0x0
r9     0x0
r10    0x1cb3
r11    0x246
r12    0x7fff50088718
r13    0x1
r14    0xc9086e0
r15    0x3
rip    0x2e9de2e
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
tls.(*Conn).readRecordOrCCS(0xc000fec388, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc001701c60 sp=0xc0017019e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000fec388, {0xc0013d4000, 0x1000, 0xc000f0a6c0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc001701cc8 sp=0xc001701c60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc0012d1200, {0xc0013d4000?, 0x32dcb05?, 0x9219480?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc001701d28 sp=0xc001701cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc000f08780)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc001701d60 sp=0xc001701d28 pc=0x3084943
bufio.(*Reader).Peek(0xc000f08780, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc001701d80 sp=0xc001701d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0012d1200)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc001701fc8 sp=0xc001701d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc001701fe0 sp=0xc001701fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001701fe8 sp=0xc001701fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 227
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 236 gp=0xc001102e00 m=nil [select]:
runtime.gopark(0xc0013adf48?, 0x2?, 0xc0?, 0xdd?, 0xc0013adef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0013add78 sp=0xc0013add58 pc=0x2f1e50e
runtime.selectgo(0xc0013adf48, 0xc0013adef0, 0xc000f96bc0?, 0x0, 0x0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc0013adeb8 sp=0xc0013add78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc0012d1200)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc0013adfc8 sp=0xc0013adeb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc0013adfe0 sp=0xc0013adfc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0013adfe8 sp=0xc0013adfe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 227
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 193 gp=0xc00064ba40 m=nil [sync.WaitGroup.Wait]:
runtime.gopark(0xc00149ba60?, 0x2eb5874?, 0x0?, 0xa3?, 0x7fe57c39bb80?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc00149ba00 sp=0xc00149b9e0 pc=0x2f1e50e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc000f974d0, 0x0, 0x1, 0x0, 0x19)
	/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc00149ba68 sp=0xc00149ba00 pc=0x2efa59d
sync.runtime_SemacquireWaitGroup(0xc000013d88?, 0x40?)
	/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc00149baa0 sp=0xc00149ba68 pc=0x2f1fc2e
sync.(*WaitGroup).Wait(0xc000f974c8)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc00149bac8 sp=0xc00149baa0 pc=0x2f43405
golang.org/x/sync/errgroup.(*Group).Wait(0xc000f974c0)
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc00149bae8 sp=0xc00149bac8 pc=0x4a4bcde
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610({0x7fe57d449c60, 0xc000652600}, 0xc000bf6d38, {0xc00092fab8, 0x12}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:525 +0x8de fp=0xc00149bd90 sp=0xc00149bae8 pc=0x6c31fbe
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000e56000, {0x7fe57d449c60, 0xc000652600}, {0xc00092fab8, 0x12}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:539 +0x29d fp=0xc00149bf08 sp=0xc00149bd90 pc=0x6c549fd
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7fe57d449c60?, 0xc000652600?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:362 +0x151 fp=0xc00149bfc0 sp=0xc00149bf08 pc=0x6c52a91
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x28 fp=0xc00149bfe0 sp=0xc00149bfc0 pc=0x6c52908
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00149bfe8 sp=0xc00149bfe0 pc=0x2f26cc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 226
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:355 +0x98

rax    0xfffffffffffffffc
rbx    0x5
rcx    0x2e9de2e
rdx    0x80
rdi    0x5
rsi    0x7fff500886d4
rbp    0x7fff50088688
rsp    0x7fff50088640
r8     0x0
r9     0x0
r10    0x1cb3
r11    0x246
r12    0x7fff50088718
r13    0x1
r14    0xc9086e0
r15    0x3
rip    0x2e9de2e
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068004614032068608
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e733.33%7 of 21 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 22none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-vc5p5t" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00160ec60>: 
    failed to create HCP cluster cluster-zstream-4-22-vc5p5t, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-vc5p5t in resource group rg-zstream-upgrade-4-22-xzbf97, error: failed waiting for cluster="cluster-zstream-4-22-vc5p5t" in resourcegroup="rg-zstream-upgrade-4-22-xzbf97" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-vc5p5t in resource group rg-zstream-upgrade-4-22-xzbf97, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-vc5p5t" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00160ec60>: 
    failed to create HCP cluster cluster-zstream-4-22-vc5p5t, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-vc5p5t in resource group rg-zstream-upgrade-4-22-xzbf97, error: failed waiting for cluster="cluster-zstream-4-22-vc5p5t" in resourcegroup="rg-zstream-upgrade-4-22-xzbf97" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-vc5p5t in resource group rg-zstream-upgrade-4-22-xzbf97, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-qkdlc9" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00157e5a0>: 
    failed to create HCP cluster cluster-zstream-4-22-qkdlc9, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-qkdlc9 in resource group rg-zstream-upgrade-4-22-2vwndt, error: failed waiting for cluster="cluster-zstream-4-22-qkdlc9" in resourcegroup="rg-zstream-upgrade-4-22-2vwndt" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-qkdlc9 in resource group rg-zstream-upgrade-4-22-2vwndt, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-qkdlc9" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00157e5a0>: 
    failed to create HCP cluster cluster-zstream-4-22-qkdlc9, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-qkdlc9 in resource group rg-zstream-upgrade-4-22-2vwndt, error: failed waiting for cluster="cluster-zstream-4-22-qkdlc9" in resourcegroup="rg-zstream-upgrade-4-22-2vwndt" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-qkdlc9 in resource group rg-zstream-upgrade-4-22-2vwndt, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapErrors | 0xc0011128a0>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-m5mrms, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-m5mrms" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-m5mrms, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapErrors | 0xc0011128a0>: 
    failed to create HCP cluster cilium-cl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-m5mrms, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-m5mrms" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-m5mrms, error: context deadline exceeded
    ...
occurred
Contributing tests (17)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.02
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
Affected runs (7)
Date (UTC)Associated PRProw job
n/a2068082178301890560
n/a2068063774274752512
n/a2068063035146113024
n/a2068004614032068608
n/a2067949798622760960
n/a2067929498837323776
n/a2067898279315640320
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e523.81%5 of 21 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 7none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:126]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000be83c0>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-l9sf6s, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-l9sf6s" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:126]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000be83c0>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-l9sf6s, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-l9sf6s" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:484]: failed to create nodepool nps-420-422 at version 4.20.26
Unexpected error:
    <*fmt.wrapErrors | 0xc000980b40>: 
    failed to create NodePool nps-420-422, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nps-420-422 in resource group rg-np-skip-minor-b69dlq-gjbzkc, error: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-b69dlq" in resourcegroup="rg-np-skip-minor-b69dlq-gjbzkc" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:484]: failed to create nodepool nps-420-422 at version 4.20.26
Unexpected error:
    <*fmt.wrapErrors | 0xc000980b40>: 
    failed to create NodePool nps-420-422, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nps-420-422 in resource group rg-np-skip-minor-b69dlq-gjbzkc, error: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-b69dlq" in resourcegroup="rg-np-skip-minor-b69dlq-gjbzkc" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:138]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc000948600>: 
    failed to create NodePool np-2, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-2 in resource group cx-rg-cluster-x5k42b, error: failed waiting for nodepool="np-2" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-x5k42b" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:138]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc000948600>: 
    failed to create NodePool np-2, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-2 in resource group cx-rg-cluster-x5k42b, error: failed waiting for nodepool="np-2" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-x5k42b" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
Affected runs (5)
Date (UTC)Associated PRProw job
n/a2068004614032068608
n/a2067949798622760960
n/a2067929498837323776
n/a2067889808788688896
n/a2067779867528663040
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
full failure pattern:
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e314.29%3 of 21 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 10none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-bm6qd9"
Unexpected error:
    <*fmt.wrapError | 0xc000702420>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-76lgc.apps.aro.n6x4q8i7r6l8a4d.ldfc.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-76lgc.apps.aro.n6x4q8i7r6l8a4d.ldfc.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-bm6qd9"
Unexpected error:
    <*fmt.wrapError | 0xc000702420>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-76lgc.apps.aro.n6x4q8i7r6l8a4d.ldfc.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-76lgc.apps.aro.n6x4q8i7r6l8a4d.ldfc.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-kmdgjv"
Unexpected error:
    <*fmt.wrapError | 0xc0009ea4c0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-pvsjg.apps.aro.z1b7r2v4p5j6g6y.dxfv.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-pvsjg.apps.aro.z1b7r2v4p5j6g6y.dxfv.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-kmdgjv"
Unexpected error:
    <*fmt.wrapError | 0xc0009ea4c0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-pvsjg.apps.aro.z1b7r2v4p5j6g6y.dxfv.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-pvsjg.apps.aro.z1b7r2v4p5j6g6y.dxfv.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-nlr5hm"
Unexpected error:
    <*fmt.wrapError | 0xc000f0a2a0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-2hbpj.apps.aro.m6l1q1f2b5a8o3l.la4x.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-2hbpj.apps.aro.m6l1q1f2b5a8o3l.la4x.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-nlr5hm"
Unexpected error:
    <*fmt.wrapError | 0xc000f0a2a0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-2hbpj.apps.aro.m6l1q1f2b5a8o3l.la4x.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-2hbpj.apps.aro.m6l1q1f2b5a8o3l.la4x.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.203
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.192
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2068082178301890560
n/a2068004614032068608
n/a2067949798622760960
Interrupted by User
full failure pattern:
Interrupted by User
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
e2e314.29%3 of 21 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 1 · Jun 18: 0 · Jun 19: 3none
Full failure examples (1)
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2068082178301890560
n/a2068004614032068608
n/a2067929498837323776
cluster has not yet reached Succeeded state
full failure pattern:
cluster has not yet reached Succeeded state
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e29.52%2 of 21 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 1 · Jun 18: 0 · Jun 19: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.008s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.008s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068063035146113024
n/a2067929498837323776
Cluster should become ready within '20.000000' minutes
full failure pattern:
Cluster should become ready within '20.000000' minutes
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e29.52%2 of 21 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 2none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.005s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.005s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068063035146113024
n/a2067929498837323776
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
full failure pattern:
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e29.52%2 of 21 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:191]: failed to create node pool version 4.19.7 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc001776480>: 
    failed waiting for nodepool="np-ver-4-19-7" for cluster "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-9r69jz" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:191]: failed to create node pool version 4.19.7 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc001776480>: 
    failed waiting for nodepool="np-ver-4-19-7" for cluster "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-9r69jz" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:138]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc001027260>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-cnkjd2" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:138]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc001027260>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-cnkjd2" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068082178301890560
n/a2067949798622760960
failed to wait for first cluster "<cluster>" to complete creation (timeout '20.000000' minutes)
full failure pattern:
failed to wait for first cluster "<cluster>" to complete creation (timeout '20.000000' minutes)
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e29.52%2 of 21 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 2none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068063035146113024
n/a2067929498837323776
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...
full failure pattern:
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 2 · Jun 18: 0 · Jun 19: 8none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012fe150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-6b2hnp" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0012fe150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-6b2hnp" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001632678>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-k7ntz6" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001632678>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-k7ntz6" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00105e5e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-bfr57d" in resourcegroup="rg-np-version-upgrade-bfr57d-2zkjv8" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc00105e5e8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-bfr57d" in resourcegroup="rg-np-version-upgrade-bfr57d-2zkjv8" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
Contributing tests (8)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067929498837323776
VerifySimpleWebApp failed: route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
full failure pattern:
VerifySimpleWebApp failed: route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001c2a060>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-gpqm7.apps.aro.cilium-cluster.95la.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-gpqm7.apps.aro.cilium-cluster.95la.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001c2a060>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-gpqm7.apps.aro.cilium-cluster.95la.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-gpqm7.apps.aro.cilium-cluster.95la.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001900270>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-kt5nq.apps.aro.cilium-cl.i3eu.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-kt5nq.apps.aro.cilium-cl.i3eu.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc001900270>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-kt5nq.apps.aro.cilium-cl.i3eu.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-kt5nq.apps.aro.cilium-cl.i3eu.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068082178301890560
tls: failed to verify certificate: x509: certificate signed by unknown authority
full failure pattern:
tls: failed to verify certificate: x509: certificate signed by unknown authority
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-4rkx7j"
Unexpected error:
    <*fmt.wrapError | 0xc000e0a620>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-9rwln.apps.aro.s0y5l7a1j6m3k3z.r0ib.australiaeast.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-4rkx7j"
Unexpected error:
    <*fmt.wrapError | 0xc000e0a620>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-9rwln.apps.aro.s0y5l7a1j6m3k3z.r0ib.australiaeast.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068063774274752512
node pool <nodepool> verification failed: <node> (release images unchanged: [quay
full failure pattern:
node pool <nodepool> verification failed: <node> (release images unchanged: [quay
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:220]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.errorString | 0xc00081af90>: 
    node pool upgrade verification failed: e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk (release images unchanged: [quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258 quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f quay.io/openshift/origin-insights-runtime-extractor:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13 quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184 quay.io/openshift/origin-kube-rbac-proxy:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9 arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1 quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef quay.io/openshift/origin-insights-runtime-exporter:latest]); e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq (not ready); nodes=[{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk","ready":true,"containerRuntimeVersion":"cri-o://1.34.2-2.rhaos4.21.gitc8e8b46.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-4xsj8","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44c807272cb410a84d347dda34dd77990697384e450b7b832e533e2a2352de76","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48c6e2725865ef2b8b7d6dc501c1778d681ed1eaf8bccad0d978c163b36a8ad9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad2768f4eed20ecddef9652e72c2536a9c5a129a733ccb8c423b8421b1d047bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd7c7b127ef0c7067d52f4ae8f3f95f878ec1c8b78a64fbb89f184824211ed80","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1110f6e8d66ba68224f9a7eedf1baa54de2d0c58332f611862242fe8ff311295","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1459af715cdfb7e21e056d793478f4387d077ad3d4d4e095c250c10eef809631","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29dd11a9e6e7344676611823c362862070b2ebc080793514aa14178527bfa7f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f35aba18e4a81d2e301162430d451d147378a5bd74a711135bead02921256554","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e18b7e1a7e67c4cff5e27d62c7de71968108b4366f8e30619fe6d277288dd63","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b61249fb7f26b253b8e634ef8cc47aa599b108e74651dba2c0dbda94df82f30","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab245b1c592c3179b63e3a79fa641f8125c16a29a67338f9adb489fb85af87b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cd91d2f0e56a7e3484ad8718845f9747c94180b0ee975fb62ffc75df214577b","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16139e0510e8b95cab7c2645e63f08d069c15ed64553e5133b99e59736ead4df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84113d4695be785283aacfc42fb9905bb7cd9233c4157f7d09726e641ea0d648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:761ccd3ced18e3e3d3760ece8e13bc07eb6119fdc05e4e5f7677d2695ab99ff4","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d6207a89a90d9f3408da1ec88e8c13c8c23330faf4c82be14c1e07b9ddb44a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-9txmg","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00f6ba985494302f2d431ad27b9bc6300d0c0d2dd0433e6403525e427a3341fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5469f5ec20ac2e7a2d82e84767c1c635671fce4ae3d7c27402289254f1183e04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a127e2a436b84937d2d970f3c316d82f61da498f04ec6036a91ad4284403f913","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d69fdb4df0385545fea609e6665de6c4b6dd4763b73a35a9d49456f879987925","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84493322067d59517be692eb5b1ffc90ae87f1a382459de635a69db4520e2750","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0476037c9c3b76a0135bc565733743cda8777479dc92a5cf2e6b79e999d35e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq","ready":false,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678"]}]
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:220]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.errorString | 0xc00081af90>: 
    node pool upgrade verification failed: e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk (release images unchanged: [quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258 quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f quay.io/openshift/origin-insights-runtime-extractor:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13 quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184 quay.io/openshift/origin-kube-rbac-proxy:latest quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9 arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1 quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef quay.io/openshift/origin-insights-runtime-exporter:latest]); e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq (not ready); nodes=[{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-7csfx-m9ggk","ready":true,"containerRuntimeVersion":"cri-o://1.34.2-2.rhaos4.21.gitc8e8b46.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8e18e0ae21b3b0179b9df6930b28609d89bb789ac6fe2a681abae95e44f91c24","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9427d2eed5cf78be6cf704503c23f8adff2f3faddacbd90d958f4c041a792a71","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b64f2faba9bedd3c3b9814fc72d69cf92a4fc06119c20b9b5a21f80ccb921b2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f25e24a7d7292e91b0daeb2272bb8443d39ee048733199ff60aabbb33111bdaa","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e5434d7dbb89536ee14f4e186599d94e3812b8fa46551d3d9e22c799a604b5d1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f3968be7c4898d66afca3704a3e5b22974cbfd00992edd7be6cf17c06d284360","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:697e6f65c6560fdd89dcdafdfc40b11f4bd101d2b61ede23d4a622e8ff7c9257","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:830a6e3b4f3a6290696bafb4ba1d9038684dd0db54ddc44111a26de0a3693b50","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-4xsj8","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:44c807272cb410a84d347dda34dd77990697384e450b7b832e533e2a2352de76","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:48c6e2725865ef2b8b7d6dc501c1778d681ed1eaf8bccad0d978c163b36a8ad9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ad2768f4eed20ecddef9652e72c2536a9c5a129a733ccb8c423b8421b1d047bd","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:cd7c7b127ef0c7067d52f4ae8f3f95f878ec1c8b78a64fbb89f184824211ed80","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1110f6e8d66ba68224f9a7eedf1baa54de2d0c58332f611862242fe8ff311295","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1459af715cdfb7e21e056d793478f4387d077ad3d4d4e095c250c10eef809631","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:29dd11a9e6e7344676611823c362862070b2ebc080793514aa14178527bfa7f1","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f35aba18e4a81d2e301162430d451d147378a5bd74a711135bead02921256554","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2e18b7e1a7e67c4cff5e27d62c7de71968108b4366f8e30619fe6d277288dd63","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9b61249fb7f26b253b8e634ef8cc47aa599b108e74651dba2c0dbda94df82f30","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1ab245b1c592c3179b63e3a79fa641f8125c16a29a67338f9adb489fb85af87b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cd91d2f0e56a7e3484ad8718845f9747c94180b0ee975fb62ffc75df214577b","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:16139e0510e8b95cab7c2645e63f08d069c15ed64553e5133b99e59736ead4df","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84113d4695be785283aacfc42fb9905bb7cd9233c4157f7d09726e641ea0d648","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:761ccd3ced18e3e3d3760ece8e13bc07eb6119fdc05e4e5f7677d2695ab99ff4","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d5d6207a89a90d9f3408da1ec88e8c13c8c23330faf4c82be14c1e07b9ddb44a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-9txmg","ready":true,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b7704b7459123ea0ffd972f12e504dc293043c8a434663944018471c127b83fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d40579aa08dd4b1af294eec1d0a6b64763a177813574a7ebb0786adeebb760bb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f956e3661b3f0f0155c9685c22715539f5f46b2c9c844998bbfe03e153d80e0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e4ec9dbacf37f1d012121164b0f8930e749b921439fcfe0d0c6938d2ec026f8b","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:865f191455cdeb365947117d2ce045720652bd414a8bb3b95f2c7696e2665abe","arohcpocpprod.azurecr.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e2f3681e052f711b6c064657771c7be246be411cb7f5d8965ac3ee8fd96c84ac","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1514ad8d2566331d0266596ad2718e15b156bed04ad0e089b3d1d2107a12208f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:97b7f9c8d69dd781d2b3bcec0e2f2a8e53f0f2bf39a56ddaecf3d97efe94b8a6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4189cf70bef60f627e2bd42c7791097b89661415c51e39d480666f2f6aa05cf0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9157d87c55920f81b4b2a5792fed59a2707e64bed22c737fc517dc64e48964","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:81f833a8ac1a6ad01c960bc2273b99f216a06ff44e5d99606e6f858c5962ec3b","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fe705e0a95d7460dcbb255989876a8cdb8f2064faad5e5b89fb89ca05f7d43c5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:387e994a39948184388421bc4e2bfdfd604ebb18b0e3fbb209b256ac03cc944f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5da788fada2dd262923584ba23b1e0151be15be754bb16cc94e7716cfef35917","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5e9e2ec5df31d1d37c3faba01952b21b974a3dce50a6bba6b35dd450d1f75fd7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd545c74b9b744796b7e4d02de3647b0168f2116d804745be09c5b10170500d3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6182d7fc5acc733c1d1ce4732f2c38cca22c2e5423c95f7ad1c68ef31b6ed7a2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:70f9c4389456eb1840df588af1e02f7c1a4b21b6a38cd78c006a867509ba2841","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0d4670da10463670c429378c06d1a77289eada6107e5f527862e3ccaab20f482","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f9cdbcb6d3b6d86a7652288ace1acda2e69f4daeaf525159b3c65d2fe948e487","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:75daf2f2d96d1122e7f6b8fcab558c5d147d812e418107c0b2c16d1b80aa8c59","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98eea0ff366a94c51cfc0d5fa04e308db59e13a43a62f9a4ff0f82794f8ecabf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift/origin-insights-runtime-extractor@sha256:3fb27467f09b64cff3f06d77ef4a894bd278bbec6cebbbb4c32692348721e258","quay.io/openshift/origin-insights-runtime-extractor@sha256:e25f086d116016c008a7dd316f895afe8296b6d3232ea06bcbaa5a614e9c762f","quay.io/openshift/origin-insights-runtime-extractor:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:43ce498501cc7bfe458890bb6722dc3a4dbeddf65996137aa4f1b1c66b250dc3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ccb2cd623189b25816f45e2b808b6e247a15f0e91e4c79d1536764175592e491","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift/origin-kube-rbac-proxy@sha256:761a0aaf39f7076c7333760023ba0832472e9164f3a2ff9df79fa8fd92c38aea","quay.io/openshift/origin-kube-rbac-proxy@sha256:c2b4bb716a14c5df2be3237ade5f36091bc0dd9a0d72f78dc7bd26ba8ca74184","quay.io/openshift/origin-kube-rbac-proxy:latest","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0cb71fabdd9fd54fb00003c6e4d39c9d89952d6286eaf7944fdd1bec94434a40","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f37f2309807dd6fa95f93965bbd2f7bc61a1279f57f9c60a6c55e4269e765fa4","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:17513a60fba837ca47e269dabddf9f814f2f51ceff0d5db1b03b535032de881f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c24d6b5b9c632eea98420d6c5497e6e275eae3fc8b94b672618c39f298d347eb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00f6ba985494302f2d431ad27b9bc6300d0c0d2dd0433e6403525e427a3341fc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5469f5ec20ac2e7a2d82e84767c1c635671fce4ae3d7c27402289254f1183e04","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a127e2a436b84937d2d970f3c316d82f61da498f04ec6036a91ad4284403f913","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d69fdb4df0385545fea609e6665de6c4b6dd4763b73a35a9d49456f879987925","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3052b5189a3ddc4874ccfdb910f2dc5e529ff49c113d97fb265e6ef600be74da","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eda11e652e4e330f3cdd881504b4886fec1c7efb74be88252e3ac906a5964044","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0070f13fd8daaa7cf9d32b35473f8df38b2eb6c8c1d157175a6bd2289495cc","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2cfba758f3448568e6f844a2c380de62c2d84d2d21c7033a91bab994994fb28d","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:78346c124edee23c689cdaa0c6446730e012676188556449ae87f8ad55013de7","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d342db4864c59bc712b6d7342da9e21af080d531806461371c17fca360562934","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:405226e2d610843087ac413348fb6677371c4aa7f7c9e6414374c8da7dd359e6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:76c82141d914b283b7195b46ef0ee3bb857cb9a8fdf4383c69fa0ac447cf6f69","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:da0074c5f34a9c29f87d6133adcfc28d1d3d130158b71d1256fa4f1fa6bc7df6","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:df57deaad79cd1370a73a6171cae724b1a81cfe30c06c3347caf8311fdd80132","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:84493322067d59517be692eb5b1ffc90ae87f1a382459de635a69db4520e2750","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e0476037c9c3b76a0135bc565733743cda8777479dc92a5cf2e6b79e999d35e8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:0ec84c1ff3cc519a88983126502b5259b81eeb23a81b7e8f14ad67905c8f2ece","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ab88f2c42768ee3d62fe458c851cde4723159f2cb54fc423e0ef9d9dbffae22e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:3bcece71d1464a21eda05d57b252fbc67106f47e3c406c4217d248cdd84f7e75","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5ca2ec2c86d1bb9ae617ec423465cac83fd0a45a43cf7daf10e59abe7a40b7e9","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6f9f08966fe12ea19f8209124da71f43bae0b096318c0f473381533b17b2e4e9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d60074e3eea17ff994da59428597a96285692d7a06b6a1efcd77431b99626e2e","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:05838d8f470015aeb5aa21a73b6c031416e623f18e525d4e7331115496375cc5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a26445905b6e13499cb88d4f520c01299a7533e57c7da97187e2aaf80cfba876","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:13e554d6c59a66e19f41be483e972de804b5ced2af9dfe24898a7b7dfcc03ceb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b0ea76f0c1f315055337ac69d303c8731058f76be8d68a6c9511465c09d379fb","quay.io/openshift/origin-insights-runtime-exporter@sha256:403c19989c47cc8e3260781ec3303cbb604f73ad8e12a3b6917febccc65bf7e1","quay.io/openshift/origin-insights-runtime-exporter@sha256:97c435176cd7e4330e1ec5a311d5de84c15b8f3af5bde35769e6bfeb8cbdf3ef","quay.io/openshift/origin-insights-runtime-exporter:latest"]},{"name":"e1f6a5i1k2y4w8q-npupgrade-4-21-n8lxx-pbwrq","ready":false,"containerRuntimeVersion":"cri-o://1.34.3-2.rhaos4.21.gitd80d17d.el9","releaseImages":["quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1e53c03a51a59765d42057649fe3a66bddb52c94c726d2b95ff0e5c15e1d6cad","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:d3de06fcebb47c51068da5176f97301ac635999c655c22bd6885ddf8b3aad625","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:71280e82ac402932d36c7adaa376222e09b69bbebff92435da8a8b969ed56c1f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ffbbc5825c670e51dd40ad2ad18c30e35c4ed0881c321e24c1e444351936c1b3","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:00a04772c6c6be5afad4047c60611f6a203c46d82e03a2d997571e863b82fc0f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:88480e1088fc09c62e7fdadbc913ec47ba108c981d9c2b268ed88391c161e0bf","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a1f6e54de0ca1c65797c282ec6cf1ac825e09fe95a73f4db0ba329ef510522f0","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bd58d17a667c14b9ce0ec72a5c2dcb8befe01984d9530387a0ba81e261ac79be","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:806f59bfd2bd2a16783e8a0d0c0fb781f19d26083dc8846fd9b05064bd9f8ecb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c8945cfb5c06ce3c0a6b0af2ee6c833a14c2a51c05901a8f18aec232e1705567","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:5d855ae8c7a2a1b8661f0ac7020414b689a4a0f44620a2279557bd1205ab9e4f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:af5b3de12122f266ee26b6f2281e6527b4a6b43f76a4e1b0570e9a68c3b1df9a","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8000255d2ca906a290d03a70eec7f97eec8b59ae06718d73291bea5913b23ab8","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c20ade2cb42d7de18d9a657904705da8419c3f02c08b21be52dd5a2a0e26c729","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4cbf3254c70b2b9b2c11094f8a448d0c731b535f0a82c4243c6377f13f19aea5","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b269b105932a07f4500815c00321283ba5478a91e014f98389c3f252960dd849","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2a795f0788e8b65cf35ac9b3f0728088d071b9d671630cc46185f4fbbea9fa0c","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b692752299fea9e227366fb41594fade2cd560a2b1760741cab9778ee7f54f8f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:06a4bad675ec675cb95004919525dccb8c5b6e3c196a53a4a016297236c92a5f","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b4815eab1f5bc56ce09cb4b90407f96eb21ac3783986eb62938d318a09d71753","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2b54a38911811d54033561dfdcfcaca869d890c857bace8f0822ab6271430fa9","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b375876c5a637e0ff0ef10c04595e93421426190d6a9649f97b8052f2a8b5124","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:92ac9c923140e97fd9aac51307774936b6b53c0ddd54c570249e6d9b37c6a424","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aeee2eb730d93ff495c22b5381fcabb76ab70e06ad241b56237103eb6a8a4e05","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:6a82f6d2155134ec31688a2374b2a4d4096fa462b948cc06d66b7b14bf2c4ad2","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:769a13b88ac0947ae0d77a07074af4b56d609034fca540534954ad2e8711dacb","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2aa1cec184f046c9629bba43f183d533adf13ead8de7e4b6bcfe3c1d16662912","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4d8e72000fedb2e238aecf2bd829b32ada7d5f8746055fd866eb30484eb12d13","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:2ab7efa378fa0f0c54b40369926627c9f9909ba5e5a18738df05190029f99c15","quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a977f08616d2c9ee5813118673180f2dbf26ec6178b795e38833a79862611901","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:03ff3448f5bb02f3a255eccd3c26730f65fdee658a5efb28ecad7e539bed9a8c","arohcpocpprod.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678"]}]
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067820806943019008
x509: certificate signed by unknown authority
full failure pattern:
x509: certificate signed by unknown authority
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 1 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.003s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.003s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <x509.UnknownAuthorityError>: 
    x509: certificate signed by unknown authority
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067820806943019008
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
full failure pattern:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:406 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 600.001s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:406 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False) ingress (Available=False)]
Expected
    <[]string | len:2, cap:2>: ...
to be empty
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067949798622760960
ERROR CODE: InternalServerError; detail message Get "<url>": net/http: TLS handshake timeout; provider Microsoft.Compute
full failure pattern:
ERROR CODE: InternalServerError; detail message Get "<url>": net/http: TLS handshake timeout; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:168]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc001910860>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-td4tgf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "name": "56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "status": "Failed",
      "startTime": "2026-06-19T18:20:56.860797749Z",
      "endTime": "2026-06-19T18:22:33.081221493Z",
      "error": {
        "code": "InternalServerError",
        "message": "Get \"https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus?%24filter=location+eq+%27brazilsouth%27\u0026api-version=2021-07-01\": net/http: TLS handshake timeout"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:168]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc001910860>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-td4tgf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "name": "56c7b5c9-051e-4ae5-b74b-cfc0e7d6c9e0",
      "status": "Failed",
      "startTime": "2026-06-19T18:20:56.860797749Z",
      "endTime": "2026-06-19T18:22:33.081221493Z",
      "error": {
        "code": "InternalServerError",
        "message": "Get \"https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus?%24filter=location+eq+%27brazilsouth%27\u0026api-version=2021-07-01\": net/http: TLS handshake timeout"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068004614032068608
Get "<url>": dial tcp <ip>:<port>: i/o timeout
full failure pattern:
Get "<url>": dial tcp <ip>:<port>: i/o timeout
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to get ClusterVersion resource
Unexpected error:
    <*url.Error | 0xc000b63410>: 
    Get "https://api.i2o1x2r3h8b1j4g.mbnd.brazilsouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to get ClusterVersion resource
Unexpected error:
    <*url.Error | 0xc000b63410>: 
    Get "https://api.i2o1x2r3h8b1j4g.mbnd.brazilsouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2067949798622760960
Get "<url>": net/http: TLS handshake timeout
full failure pattern:
Get "<url>": net/http: TLS handshake timeout
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:162]: failed to list nodes after AZ nodepool creation
Unexpected error:
    <*url.Error | 0xc0016e2240>: 
    Get "https://api.f2e6d6p6j1q8o4d.91nt.uksouth.aroapp-hcp.io:443/api/v1/nodes": net/http: TLS handshake timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:162]: failed to list nodes after AZ nodepool creation
Unexpected error:
    <*url.Error | 0xc0016e2240>: 
    Get "https://api.f2e6d6p6j1q8o4d.91nt.uksouth.aroapp-hcp.io:443/api/v1/nodes": net/http: TLS handshake timeout
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068106661532274688
Post "<url>": dial tcp <ip>:<port>: i/o timeout
full failure pattern:
Post "<url>": dial tcp <ip>:<port>: i/o timeout
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:236]: should be able to create SelfSubjectReview for admin credential 1
Unexpected error:
    <*url.Error | 0xc00130c030>: 
    Post "https://api.u6k7t4d6d9b4a5c.vf44.brazilsouth.aroapp-hcp.io:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:236]: should be able to create SelfSubjectReview for admin credential 1
Unexpected error:
    <*url.Error | 0xc00130c030>: 
    Post "https://api.u6k7t4d6d9b4a5c.vf44.brazilsouth.aroapp-hcp.io:443/apis/authentication.k8s.io/v1/selfsubjectreviews": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068082178301890560
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Pro...
full failure pattern:
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc00101e000>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc00101e000>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068082178301890560
dial tcp: lookup console-openshift-console
full failure pattern:
dial tcp: lookup console-openshift-console
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e14.76%1 of 21 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.006s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc000bcba90>: 
    dial tcp: lookup console-openshift-console.apps.aro.b0n9v3c0r2b5h8v.2xda.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.006s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc000bcba90>: 
    dial tcp: lookup console-openshift-console.apps.aro.b0n9v3c0r2b5h8v.2xda.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068082178301890560