CIHealth
<
Single Day: Jul 8▾
Last 7 Days Weekly: Jul 6 - Jul 12 Sprint: Jul 6 - Jul 19 Single Day: Jul 8
Custom time window
> Reset
View JSON API

Environment: PROD

Failure patterns: 13 · Failures matched: 64 · Total runs: 4 · Runs affected: 3

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Deserialization Error: no output from command
full failure pattern:
Deserialization Error: no output from command
Signal: NoiseSignal: Noise — source deserialization/no-output error
e2e125.00%1 of 4 job runs affectedNoiseSignal: Noise — source deserialization/no-output errorShow trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 3none
Full failure examples (3)
c000ea0748)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc00158dac8 sp=0xc00158daa0 pc=0x2f5b405
golang.org/x/sync/errgroup.(*Group).Wait(0xc000ea0740)
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc00158dae8 sp=0xc00158dac8 pc=0x4a8aabe
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610({0x7f130446f260, 0xc00157e930}, 0xc000a2af30, {0xc000dc6ea0, 0x22}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:525 +0x8de fp=0xc00158dd90 sp=0xc00158dae8 pc=0x6c8efde
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000e4c840, {0x7f130446f260, 0xc00157e930}, {0xc000dc6ea0, 0x22}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:537 +0x29d fp=0xc00158df08 sp=0xc00158dd90 pc=0x6cb215d
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7f130446f260?, 0xc00157e930?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:360 +0x151 fp=0xc00158dfc0 sp=0xc00158df08 pc=0x6cb01f1
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:369 +0x28 fp=0xc00158dfe0 sp=0xc00158dfc0 pc=0x6cb0068
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00158dfe8 sp=0xc00158dfe0 pc=0x2f3ecc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 517
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353 +0x98

goroutine 533 gp=0xc001103340 m=nil [select]:
runtime.gopark(0xc00189d768?, 0x2?, 0x0?, 0x40?, 0xc00189d75c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc00189d5e8 sp=0xc00189d5c8 pc=0x2f3650e
runtime.selectgo(0xc00189d768, 0xc00189d758, 0xc000a80180?, 0x0, 0x370521c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc00189d728 sp=0xc00189d5e8 pc=0x2f112f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x9376198, 0xc000033490}, 0x8779154?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc00189d798 sp=0xc00189d728 pc=0x3704976
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x93c8b00, {0x9376198, 0xc000033490}, 0x2f33af3)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc00189da98 sp=0xc00189d798 pc=0x3e10172
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x9376198, 0xc000033490}, 0xc000a2af30, {0xc000dc6ea0, 0x22}, {0xc0008e1ca0, 0xd})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:240 +0x365 fp=0xc00189dc20 sp=0xc00189da98 pc=0x6c8bf85
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x9376198?, 0xc000033490?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x72 fp=0xc00189dcc8 sp=0xc00189dc20 pc=0x6c8b9b2
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x9376198?, 0xc000033490?}, 0xc0000334a0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc00189dd20 sp=0xc00189dcc8 pc=0x36b7f42
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x9376198, 0xc000033490}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc00189de68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc00189dd98 sp=0xc00189dd20 pc=0x36b5fee
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x9376128, 0xc000e91b80}, 0xc000a2af30, {0xc000dc6ea0, 0x22}, {0xc0008e1ca0, 0xd}, 0x274a48a7800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:196 +0x285 fp=0xc00189def0 sp=0xc00189dd98 pc=0x6c8b725
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:522 +0x7e fp=0xc00189df78 sp=0xc00189def0 pc=0x6c8f61e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc00189dfe0 sp=0xc00189df78 pc=0x4a8ac10
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00189dfe8 sp=0xc00189dfe0 pc=0x2f3ecc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 526
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x2f40ae3
rdx    0x0
rdi    0xc9bba10
rsi    0x80
rbp    0x7ffc382442f0
rsp    0x7ffc382442a8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x3
r13    0x1
r14    0xc9b9480
r15    0x7
rip    0x2f40ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
c000ea0748)
	/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc00158dac8 sp=0xc00158daa0 pc=0x2f5b405
golang.org/x/sync/errgroup.(*Group).Wait(0xc000ea0740)
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc00158dae8 sp=0xc00158dac8 pc=0x4a8aabe
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610({0x7f130446f260, 0xc00157e930}, 0xc000a2af30, {0xc000dc6ea0, 0x22}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:525 +0x8de fp=0xc00158dd90 sp=0xc00158dae8 pc=0x6c8efde
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000e4c840, {0x7f130446f260, 0xc00157e930}, {0xc000dc6ea0, 0x22}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:537 +0x29d fp=0xc00158df08 sp=0xc00158dd90 pc=0x6cb215d
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7f130446f260?, 0xc00157e930?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:360 +0x151 fp=0xc00158dfc0 sp=0xc00158df08 pc=0x6cb01f1
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:369 +0x28 fp=0xc00158dfe0 sp=0xc00158dfc0 pc=0x6cb0068
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00158dfe8 sp=0xc00158dfe0 pc=0x2f3ecc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 517
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353 +0x98

goroutine 533 gp=0xc001103340 m=nil [select]:
runtime.gopark(0xc00189d768?, 0x2?, 0x0?, 0x40?, 0xc00189d75c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc00189d5e8 sp=0xc00189d5c8 pc=0x2f3650e
runtime.selectgo(0xc00189d768, 0xc00189d758, 0xc000a80180?, 0x0, 0x370521c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc00189d728 sp=0xc00189d5e8 pc=0x2f112f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x9376198, 0xc000033490}, 0x8779154?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc00189d798 sp=0xc00189d728 pc=0x3704976
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x93c8b00, {0x9376198, 0xc000033490}, 0x2f33af3)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc00189da98 sp=0xc00189d798 pc=0x3e10172
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x9376198, 0xc000033490}, 0xc000a2af30, {0xc000dc6ea0, 0x22}, {0xc0008e1ca0, 0xd})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:240 +0x365 fp=0xc00189dc20 sp=0xc00189da98 pc=0x6c8bf85
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x9376198?, 0xc000033490?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x72 fp=0xc00189dcc8 sp=0xc00189dc20 pc=0x6c8b9b2
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x9376198?, 0xc000033490?}, 0xc0000334a0?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc00189dd20 sp=0xc00189dcc8 pc=0x36b7f42
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x9376198, 0xc000033490}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc00189de68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc00189dd98 sp=0xc00189dd20 pc=0x36b5fee
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x9376128, 0xc000e91b80}, 0xc000a2af30, {0xc000dc6ea0, 0x22}, {0xc0008e1ca0, 0xd}, 0x274a48a7800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:196 +0x285 fp=0xc00189def0 sp=0xc00189dd98 pc=0x6c8b725
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:522 +0x7e fp=0xc00189df78 sp=0xc00189def0 pc=0x6c8f61e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc00189dfe0 sp=0xc00189df78 pc=0x4a8ac10
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00189dfe8 sp=0xc00189dfe0 pc=0x2f3ecc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 526
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x2f40ae3
rdx    0x0
rdi    0xc9bba10
rsi    0x80
rbp    0x7ffc382442f0
rsp    0x7ffc382442a8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x3
r13    0x1
r14    0xc9b9480
r15    0x7
rip    0x2f40ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
6cb01f1
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:369 +0x28 fp=0xc000ecdfe0 sp=0xc000ecdfc0 pc=0x6cb0068
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000ecdfe8 sp=0xc000ecdfe0 pc=0x2f3ecc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 282
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353 +0x98

goroutine 315 gp=0xc0016b9340 m=nil [select]:
runtime.gopark(0xc000b05f48?, 0x2?, 0xc0?, 0x5d?, 0xc000b05ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000b05d78 sp=0xc000b05d58 pc=0x2f3650e
runtime.selectgo(0xc000b05f48, 0xc000b05ef0, 0xc000e26c40?, 0x0, 0x0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000b05eb8 sp=0xc000b05d78 pc=0x2f112f7
net/http.(*persistConn).writeLoop(0xc000adefc0)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc000b05fc8 sp=0xc000b05eb8 pc=0x32fec87
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc000b05fe0 sp=0xc000b05fc8 pc=0x32fb925
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000b05fe8 sp=0xc000b05fe0 pc=0x2f3ecc1
created by net/http.(*Transport).dialConn in goroutine 283
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 314 gp=0xc0016b96c0 m=nil [IO wait]:
runtime.gopark(0xc0009cc3a0?, 0xc0000c3788?, 0xc5?, 0x3e?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0000c3768 sp=0xc0000c3748 pc=0x2f3650e
runtime.netpollblock(0x2fa0ad8?, 0x2ec2066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc0000c37a0 sp=0xc0000c3768 pc=0x2ef5fb7
internal/poll.runtime_pollWait(0x7f0ad6814a00, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc0000c37c0 sp=0xc0000c37a0 pc=0x2f34da5
internal/poll.(*pollDesc).wait(0xc001816c80?, 0xc000dec000?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000c37e8 sp=0xc0000c37c0 pc=0x2fc3907
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001816c80, {0xc000dec000, 0x1300, 0x1300})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc0000c3880 sp=0xc0000c37e8 pc=0x2fc4d39
net.(*netFD).Read(0xc001816c80, {0xc000dec000?, 0xc000dec000?, 0x5?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc0000c38c8 sp=0xc0000c3880 pc=0x31893c5
net.(*conn).Read(0xc0000b4318, {0xc000dec000?, 0x7f0ad57c8a48?, 0x7f0b1d3fb5c0?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc0000c3910 sp=0xc0000c38c8 pc=0x319b465
crypto/tls.(*atLeastReader).Read(0xc0010f29d8, {0xc000dec000?, 0x120a?, 0x2ed79d6?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc0000c3958 sp=0xc0000c3910 pc=0x3206ffb
bytes.(*Buffer).ReadFrom(0xc0009a82a8, {0x92dd8a0, 0xc0010f29d8})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc0000c39b0 sp=0xc0000c3958 pc=0x3007578
crypto/tls.(*Conn).readFromUntil(0xc0009a8008, {0x92df380, 0xc0000b4318}, 0xc0000c3a18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0000c39e8 sp=0xc0000c39b0 pc=0x32071de
crypto/tls.(*Conn).readRecordOrCCS(0xc0009a8008, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc0000c3c60 sp=0xc0000c39e8 pc=0x320425b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc0009a8008, {0xc000b0e000, 0x1000, 0xc000a4f650?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc0000c3cc8 sp=0xc0000c3c60 pc=0x320ab65
net/http.(*persistConn).Read(0xc000adefc0, {0xc000b0e000?, 0x32f4be5?, 0x92d8260?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc0000c3d28 sp=0xc0000c3cc8 pc=0x32fc4e7
bufio.(*Reader).fill(0xc000c2daa0)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc0000c3d60 sp=0xc0000c3d28 pc=0x309ca23
bufio.(*Reader).Peek(0xc000c2daa0, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc0000c3d80 sp=0xc0000c3d60 pc=0x309cbb3
net/http.(*persistConn).readLoop(0xc000adefc0)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc0000c3fc8 sp=0xc0000c3d80 pc=0x32fcff2
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc0000c3fe0 sp=0xc0000c3fc8 pc=0x32fb985
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000c3fe8 sp=0xc0000c3fe0 pc=0x2f3ecc1
created by net/http.(*Transport).dialConn in goroutine 283
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

rax    0xca
rbx    0x0
rcx    0x2f40ae3
rdx    0x0
rdi    0xc9bba10
rsi    0x80
rbp    0x7fff7dbe62d0
rsp    0x7fff7dbe6288
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x0
r14    0xc9b9480
r15    0x7
rip    0x2f40ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
6cb01f1
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:369 +0x28 fp=0xc000ecdfe0 sp=0xc000ecdfc0 pc=0x6cb0068
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000ecdfe8 sp=0xc000ecdfe0 pc=0x2f3ecc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 282
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353 +0x98

goroutine 315 gp=0xc0016b9340 m=nil [select]:
runtime.gopark(0xc000b05f48?, 0x2?, 0xc0?, 0x5d?, 0xc000b05ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000b05d78 sp=0xc000b05d58 pc=0x2f3650e
runtime.selectgo(0xc000b05f48, 0xc000b05ef0, 0xc000e26c40?, 0x0, 0x0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000b05eb8 sp=0xc000b05d78 pc=0x2f112f7
net/http.(*persistConn).writeLoop(0xc000adefc0)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc000b05fc8 sp=0xc000b05eb8 pc=0x32fec87
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc000b05fe0 sp=0xc000b05fc8 pc=0x32fb925
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000b05fe8 sp=0xc000b05fe0 pc=0x2f3ecc1
created by net/http.(*Transport).dialConn in goroutine 283
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 314 gp=0xc0016b96c0 m=nil [IO wait]:
runtime.gopark(0xc0009cc3a0?, 0xc0000c3788?, 0xc5?, 0x3e?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0000c3768 sp=0xc0000c3748 pc=0x2f3650e
runtime.netpollblock(0x2fa0ad8?, 0x2ec2066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc0000c37a0 sp=0xc0000c3768 pc=0x2ef5fb7
internal/poll.runtime_pollWait(0x7f0ad6814a00, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc0000c37c0 sp=0xc0000c37a0 pc=0x2f34da5
internal/poll.(*pollDesc).wait(0xc001816c80?, 0xc000dec000?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0000c37e8 sp=0xc0000c37c0 pc=0x2fc3907
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001816c80, {0xc000dec000, 0x1300, 0x1300})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc0000c3880 sp=0xc0000c37e8 pc=0x2fc4d39
net.(*netFD).Read(0xc001816c80, {0xc000dec000?, 0xc000dec000?, 0x5?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc0000c38c8 sp=0xc0000c3880 pc=0x31893c5
net.(*conn).Read(0xc0000b4318, {0xc000dec000?, 0x7f0ad57c8a48?, 0x7f0b1d3fb5c0?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc0000c3910 sp=0xc0000c38c8 pc=0x319b465
crypto/tls.(*atLeastReader).Read(0xc0010f29d8, {0xc000dec000?, 0x120a?, 0x2ed79d6?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc0000c3958 sp=0xc0000c3910 pc=0x3206ffb
bytes.(*Buffer).ReadFrom(0xc0009a82a8, {0x92dd8a0, 0xc0010f29d8})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc0000c39b0 sp=0xc0000c3958 pc=0x3007578
crypto/tls.(*Conn).readFromUntil(0xc0009a8008, {0x92df380, 0xc0000b4318}, 0xc0000c3a18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0000c39e8 sp=0xc0000c39b0 pc=0x32071de
crypto/tls.(*Conn).readRecordOrCCS(0xc0009a8008, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc0000c3c60 sp=0xc0000c39e8 pc=0x320425b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc0009a8008, {0xc000b0e000, 0x1000, 0xc000a4f650?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc0000c3cc8 sp=0xc0000c3c60 pc=0x320ab65
net/http.(*persistConn).Read(0xc000adefc0, {0xc000b0e000?, 0x32f4be5?, 0x92d8260?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc0000c3d28 sp=0xc0000c3cc8 pc=0x32fc4e7
bufio.(*Reader).fill(0xc000c2daa0)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc0000c3d60 sp=0xc0000c3d28 pc=0x309ca23
bufio.(*Reader).Peek(0xc000c2daa0, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc0000c3d80 sp=0xc0000c3d60 pc=0x309cbb3
net/http.(*persistConn).readLoop(0xc000adefc0)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc0000c3fc8 sp=0xc0000c3d80 pc=0x32fcff2
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc0000c3fe0 sp=0xc0000c3fc8 pc=0x32fb985
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000c3fe8 sp=0xc0000c3fe0 pc=0x2f3ecc1
created by net/http.(*Transport).dialConn in goroutine 283
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

rax    0xca
rbx    0x0
rcx    0x2f40ae3
rdx    0x0
rdi    0xc9bba10
rsi    0x80
rbp    0x7fff7dbe62d0
rsp    0x7fff7dbe6288
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x0
r13    0x0
r14    0xc9b9480
r15    0x7
rip    0x2f40ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
ync/waitgroup.go:206 +0x85 fp=0xc000e5fac8 sp=0xc000e5faa0 pc=0x2f5b405
golang.org/x/sync/errgroup.(*Group).Wait(0xc000e97840)
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc000e5fae8 sp=0xc000e5fac8 pc=0x4a8aabe
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610({0x7fc97c1bf660, 0xc000b97f50}, 0xc001082750, {0xc0009bb5c0, 0x11}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:525 +0x8de fp=0xc000e5fd90 sp=0xc000e5fae8 pc=0x6c8efde
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000c154a0, {0x7fc97c1bf660, 0xc000b97f50}, {0xc0009bb5c0, 0x11}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:537 +0x29d fp=0xc000e5ff08 sp=0xc000e5fd90 pc=0x6cb215d
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7fc97c1bf660?, 0xc000b97f50?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:360 +0x151 fp=0xc000e5ffc0 sp=0xc000e5ff08 pc=0x6cb01f1
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:369 +0x28 fp=0xc000e5ffe0 sp=0xc000e5ffc0 pc=0x6cb0068
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000e5ffe8 sp=0xc000e5ffe0 pc=0x2f3ecc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 470
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353 +0x98

goroutine 491 gp=0xc0011ae380 m=nil [select]:
runtime.gopark(0xc00165b768?, 0x2?, 0x0?, 0x5a?, 0xc00165b75c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc00165b5e8 sp=0xc00165b5c8 pc=0x2f3650e
runtime.selectgo(0xc00165b768, 0xc00165b758, 0xc0006feca0?, 0x0, 0x370521c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc00165b728 sp=0xc00165b5e8 pc=0x2f112f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x9376198, 0xc001566070}, 0x8779154?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc00165b798 sp=0xc00165b728 pc=0x3704976
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x93c8b00, {0x9376198, 0xc001566070}, 0x2f33af3)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc00165ba98 sp=0xc00165b798 pc=0x3e10172
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x9376198, 0xc001566070}, 0xc001082750, {0xc0009bb5c0, 0x11}, {0xc00154a300, 0x12})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:240 +0x365 fp=0xc00165bc20 sp=0xc00165ba98 pc=0x6c8bf85
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x9376198?, 0xc001566070?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x72 fp=0xc00165bcc8 sp=0xc00165bc20 pc=0x6c8b9b2
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x9376198?, 0xc001566070?}, 0xc001566080?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc00165bd20 sp=0xc00165bcc8 pc=0x36b7f42
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x9376198, 0xc001566070}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc00165be68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc00165bd98 sp=0xc00165bd20 pc=0x36b5fee
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x9376128, 0xc0007baff0}, 0xc001082750, {0xc0009bb5c0, 0x11}, {0xc00154a300, 0x12}, 0x274a48a7800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:196 +0x285 fp=0xc00165bef0 sp=0xc00165bd98 pc=0x6c8b725
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:522 +0x7e fp=0xc00165bf78 sp=0xc00165bef0 pc=0x6c8f61e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc00165bfe0 sp=0xc00165bf78 pc=0x4a8ac10
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00165bfe8 sp=0xc00165bfe0 pc=0x2f3ecc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 502
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xfffffffffffffffc
rbx    0x5
rcx    0x2eb5e2e
rdx    0x80
rdi    0x5
rsi    0x7ffcb9d27d24
rbp    0x7ffcb9d27cd8
rsp    0x7ffcb9d27c90
r8     0x0
r9     0x0
r10    0x393
r11    0x246
r12    0x7ffcb9d27d68
r13    0x1
r14    0xc9b9480
r15    0x3
rip    0x2eb5e2e
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
ync/waitgroup.go:206 +0x85 fp=0xc000e5fac8 sp=0xc000e5faa0 pc=0x2f5b405
golang.org/x/sync/errgroup.(*Group).Wait(0xc000e97840)
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:56 +0x1e fp=0xc000e5fae8 sp=0xc000e5fac8 pc=0x4a8aabe
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610({0x7fc97c1bf660, 0xc000b97f50}, 0xc001082750, {0xc0009bb5c0, 0x11}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:525 +0x8de fp=0xc000e5fd90 sp=0xc000e5fae8 pc=0x6c8efde
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000c154a0, {0x7fc97c1bf660, 0xc000b97f50}, {0xc0009bb5c0, 0x11}, 0x34630b8a000)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:537 +0x29d fp=0xc000e5ff08 sp=0xc000e5fd90 pc=0x6cb215d
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7fc97c1bf660?, 0xc000b97f50?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:360 +0x151 fp=0xc000e5ffc0 sp=0xc000e5ff08 pc=0x6cb01f1
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:369 +0x28 fp=0xc000e5ffe0 sp=0xc000e5ffc0 pc=0x6cb0068
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000e5ffe8 sp=0xc000e5ffe0 pc=0x2f3ecc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 470
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353 +0x98

goroutine 491 gp=0xc0011ae380 m=nil [select]:
runtime.gopark(0xc00165b768?, 0x2?, 0x0?, 0x5a?, 0xc00165b75c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc00165b5e8 sp=0xc00165b5c8 pc=0x2f3650e
runtime.selectgo(0xc00165b768, 0xc00165b758, 0xc0006feca0?, 0x0, 0x370521c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc00165b728 sp=0xc00165b5e8 pc=0x2f112f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x9376198, 0xc001566070}, 0x8779154?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc00165b798 sp=0xc00165b728 pc=0x3704976
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x93c8b00, {0x9376198, 0xc001566070}, 0x2f33af3)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc00165ba98 sp=0xc00165b798 pc=0x3e10172
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x9376198, 0xc001566070}, 0xc001082750, {0xc0009bb5c0, 0x11}, {0xc00154a300, 0x12})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:240 +0x365 fp=0xc00165bc20 sp=0xc00165ba98 pc=0x6c8bf85
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x9376198?, 0xc001566070?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x72 fp=0xc00165bcc8 sp=0xc00165bc20 pc=0x6c8b9b2
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x9376198?, 0xc001566070?}, 0xc001566080?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc00165bd20 sp=0xc00165bcc8 pc=0x36b7f42
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x9376198, 0xc001566070}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc00165be68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc00165bd98 sp=0xc00165bd20 pc=0x36b5fee
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x9376128, 0xc0007baff0}, 0xc001082750, {0xc0009bb5c0, 0x11}, {0xc00154a300, 0x12}, 0x274a48a7800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:196 +0x285 fp=0xc00165bef0 sp=0xc00165bd98 pc=0x6c8b725
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:522 +0x7e fp=0xc00165bf78 sp=0xc00165bef0 pc=0x6c8f61e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc00165bfe0 sp=0xc00165bf78 pc=0x4a8ac10
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00165bfe8 sp=0xc00165bfe0 pc=0x2f3ecc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 502
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xfffffffffffffffc
rbx    0x5
rcx    0x2eb5e2e
rdx    0x80
rdi    0x5
rsi    0x7ffcb9d27d24
rbp    0x7ffcb9d27cd8
rsp    0x7ffcb9d27c90
r8     0x0
r9     0x0
r10    0x393
r11    0x246
r12    0x7ffcb9d27d68
r13    0x1
r14    0xc9b9480
r15    0x3
rip    0x2eb5e2e
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
Contributing tests (3)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074860029835481088
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e250.00%2 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 45none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapErrors | 0xc0011863f0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-b5lpzp" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-b5lpzp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapErrors | 0xc0011863f0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-b5lpzp" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-b5lpzp, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-sh5qkl/cluster-candidate-5-0-4glj7t should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0012f0090>: 
    failed to create HCP cluster cluster-candidate-5-0-4glj7t, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-4glj7t in resource group rg-candidate-5-0-sh5qkl, error: failed waiting for cluster="cluster-candidate-5-0-4glj7t" in resourcegroup="rg-candidate-5-0-sh5qkl" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-4glj7t in resource group rg-candidate-5-0-sh5qkl, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-sh5qkl/cluster-candidate-5-0-4glj7t should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0012f0090>: 
    failed to create HCP cluster cluster-candidate-5-0-4glj7t, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-4glj7t in resource group rg-candidate-5-0-sh5qkl, error: failed waiting for cluster="cluster-candidate-5-0-4glj7t" in resourcegroup="rg-candidate-5-0-sh5qkl" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-4glj7t in resource group rg-candidate-5-0-sh5qkl, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:86]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapErrors | 0xc0010e6930>: 
    failed to create HCP cluster ea-listbk9qsn, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listbk9qsn in resource group ea-list-8krq8c, error: failed waiting for cluster="ea-listbk9qsn" in resourcegroup="ea-list-8krq8c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listbk9qsn in resource group ea-list-8krq8c, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:86]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapErrors | 0xc0010e6930>: 
    failed to create HCP cluster ea-listbk9qsn, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listbk9qsn in resource group ea-list-8krq8c, error: failed waiting for cluster="ea-listbk9qsn" in resourcegroup="ea-list-8krq8c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster ea-listbk9qsn in resource group ea-list-8krq8c, error: context deadline exceeded
    ...
occurred
Contributing tests (38)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelFIPS Mode Support with ExperimentalReleaseFeatures AFEC registered should create an HCP cluster with FIPS mode enabled via experimental tag1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.191
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2074860029835481088
n/a2074790983152701440
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:754]: failed to create nodepool npdg-421-419
Unexpected error:
    <*fmt.wrapErrors | 0xc0015aacf0>: 
    failed to create NodePool npdg-421-419, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npdg-421-419 in resource group rg-np-dg-minor-2k4w8q-d2pcfx, error: failed waiting for nodepool="npdg-421-419" for cluster "np-dg-minor-2k4w8q" in resourcegroup="rg-np-dg-minor-2k4w8q-d2pcfx" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:754]: failed to create nodepool npdg-421-419
Unexpected error:
    <*fmt.wrapErrors | 0xc0015aacf0>: 
    failed to create NodePool npdg-421-419, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool npdg-421-419 in resource group rg-np-dg-minor-2k4w8q-d2pcfx, error: failed waiting for nodepool="npdg-421-419" for cluster "np-dg-minor-2k4w8q" in resourcegroup="rg-np-dg-minor-2k4w8q-d2pcfx" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:477]: failed to create nodepool nps-420-422 at version 4.20.28
Unexpected error:
    <*fmt.wrapErrors | 0xc0024b1110>: 
    failed to create NodePool nps-420-422, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nps-420-422 in resource group rg-np-skip-minor-lfkkhh-5f9msh, error: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-lfkkhh" in resourcegroup="rg-np-skip-minor-lfkkhh-5f9msh" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:477]: failed to create nodepool nps-420-422 at version 4.20.28
Unexpected error:
    <*fmt.wrapErrors | 0xc0024b1110>: 
    failed to create NodePool nps-420-422, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool nps-420-422 in resource group rg-np-skip-minor-lfkkhh-5f9msh, error: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-lfkkhh" in resourcegroup="rg-np-skip-minor-lfkkhh-5f9msh" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:330]: failed to create nodepool npnoedge-4-20 at version 4.20.0
Unexpected error:
    <*fmt.wrapErrors | 0xc0014079e0>: 
    failed to create NodePool npnoedge-4-20, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool npnoedge-4-20 in resource group rg-np-noedge-j7lh88-lbpxgn, error: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-j7lh88" in resourcegroup="rg-np-noedge-j7lh88-lbpxgn" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:330]: failed to create nodepool npnoedge-4-20 at version 4.20.0
Unexpected error:
    <*fmt.wrapErrors | 0xc0014079e0>: 
    failed to create NodePool npnoedge-4-20, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool npnoedge-4-20 in resource group rg-np-noedge-j7lh88-lbpxgn, error: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-j7lh88" in resourcegroup="rg-np-noedge-j7lh88-lbpxgn" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.21.zLatest to 4.19.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074790983152701440
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateNodePoolAndWait timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:210]: failed to upgrade node pool npupgrade-4-20 to version 4.21.22
Unexpected error:
    <*fmt.wrapErrors | 0xc000890ea0>: 
    failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-fr5x47" resourcegroup="rg-np-version-upgrade-fr5x47-tjnxv8" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-fr5x47 in resource group rg-np-version-upgrade-fr5x47-tjnxv8, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:210]: failed to upgrade node pool npupgrade-4-20 to version 4.21.22
Unexpected error:
    <*fmt.wrapErrors | 0xc000890ea0>: 
    failed waiting for nodepool="npupgrade-4-20" in cluster="np-version-upgrade-cluster-fr5x47" resourcegroup="rg-np-version-upgrade-fr5x47-tjnxv8" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-20 in cluster np-version-upgrade-cluster-fr5x47 in resource group rg-np-version-upgrade-fr5x47-tjnxv8, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:185]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc0025850e0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-cqzk85" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-cqzk85, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:185]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc0025850e0>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-cqzk85" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-cqzk85, error: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074790983152701440
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.fips-enabled": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.fips-enabled": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 2 · Jul 5: 2 · Jul 6: 2 · Jul 7: 2 · Jul 8: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:84]: failed to create HCP cluster "fips-enabled-cluster" with FIPS enabled
Unexpected error:
    <*fmt.wrapError | 0xc0008eab80>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-c67zkg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-c67zkg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.fips-enabled\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.fips-enabled]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:84]: failed to create HCP cluster "fips-enabled-cluster" with FIPS enabled
Unexpected error:
    <*fmt.wrapError | 0xc0008eab80>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-c67zkg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-c67zkg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.fips-enabled\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.fips-enabled]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelFIPS Mode Support with ExperimentalReleaseFeatures AFEC registered should create an HCP cluster with FIPS mode enabled via experimental tag1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074674924374462464
error should indicate FIPS value must be true or false
full failure pattern:
error should indicate FIPS value must be true or false
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 2 · Jul 5: 2 · Jul 6: 2 · Jul 7: 2 · Jul 8: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:203]: error should indicate FIPS value must be true or false
Expected
    <string>: ...
to contain substring
    <string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:203]: error should indicate FIPS value must be true or false
Expected
    <string>: ...
to contain substring
    <string>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelFIPS Mode Support with ExperimentalReleaseFeatures AFEC registered should reject an HCP cluster creation with invalid FIPS tag value1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074674924374462464
cluster has not yet reached Succeeded state
full failure pattern:
cluster has not yet reached Succeeded state
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074860029835481088
Cluster should become ready within '20.000000' minutes
full failure pattern:
Cluster should become ready within '20.000000' minutes
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.004s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.004s.
Cluster should become ready within '20.000000' minutes
Expected
    <bool>: ...
to be true
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074860029835481088
failed to wait for first cluster "<cluster>" to complete creation (timeout '20.000000' minutes)
full failure pattern:
failed to wait for first cluster "<cluster>" to complete creation (timeout '20.000000' minutes)
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <context.deadlineExceededError>: 
    context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074860029835481088
VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to create FIPS check pod: pod "<pod>" is forbidden: error looking up service account <namespace>/default...
full failure pattern:
VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to create FIPS check pod: pod "<pod>" is forbidden: error looking up service account <namespace>/default: serviceaccount "default" not found
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xc000b267e0>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node e9n0y8t3z6s7y7t-np-1-6l2x9-pkqwh: failed to create FIPS check pod: pods "fips-check-" is forbidden: error looking up service account e2e-fips-test-h4zck/default: serviceaccount "default" not found
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
    <*errors.joinError | 0xc000b267e0>: 
    VerifyFIPSEnabled failed: failed to check FIPS mode on node e9n0y8t3z6s7y7t-np-1-6l2x9-pkqwh: failed to create FIPS check pod: pods "fips-check-" is forbidden: error looking up service account e2e-fips-test-h4zck/default: serviceaccount "default" not found
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelFIPS Mode Support with ExperimentalReleaseFeatures AFEC registered should create an HCP cluster with FIPS mode enabled via experimental tag1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074790983152701440
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...
full failure pattern:
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0016da1f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-cfqmg9" in resourcegroup="rg-candidate-4-21-5l8mwr" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0016da1f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-cfqmg9" in resourcegroup="rg-candidate-4-21-5l8mwr" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074790983152701440
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
full failure pattern:
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: failed to create node pool version 4.19.7 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f19d40>: 
    failed waiting for nodepool="np-ver-4-19-7" for cluster "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-jbrspk" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: failed to create node pool version 4.19.7 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f19d40>: 
    failed waiting for nodepool="np-ver-4-19-7" for cluster "cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-jbrspk" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.191
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074790983152701440
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
full failure pattern:
timeout during GetAdminRESTConfigForHCPCluster while waiting for hcpcluster creds
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e125.00%1 of 4 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJul 2: 0 · Jul 3: 0 · Jul 4: 0 · Jul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:161]: failed to get admin REST config for cluster np-version-upgrade-cluster-gv255d
Unexpected error:
    <*fmt.wrapErrors | 0xc002418720>: 
    failed waiting for hcpCluster="np-version-upgrade-cluster-gv255d" in resourcegroup="rg-np-version-upgrade-gv255d-bzslj8" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-gv255d in resource group rg-np-version-upgrade-gv255d-bzslj8, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:161]: failed to get admin REST config for cluster np-version-upgrade-cluster-gv255d
Unexpected error:
    <*fmt.wrapErrors | 0xc002418720>: 
    failed waiting for hcpCluster="np-version-upgrade-cluster-gv255d" in resourcegroup="rg-np-version-upgrade-gv255d-bzslj8" to finish getting creds, caused by: timeout '10.000000' minutes exceeded during GetAdminRESTConfigForHCPCluster for cluster np-version-upgrade-cluster-gv255d in resource group rg-np-version-upgrade-gv255d-bzslj8, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2074790983152701440