CIHealth
<
Single Day: Jun 20▾
Last 7 Days Weekly: Jun 15 - Jun 21 Sprint: Jun 8 - Jun 21 Single Day: Jun 20
Custom time window
> Reset
View JSON API

Environment: PROD

Failure patterns: 13 · Failures matched: 33 · Total runs: 13 · Runs affected: 9

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Interrupted by User
full failure pattern:
Interrupted by User
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
e2e215.38%2 of 13 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 1 · Jun 18: 0 · Jun 19: 3 · Jun 20: 2none
Full failure examples (1)
fail [:0]: Interrupted by User
fail [:0]: Interrupted by User
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068432057893130240
n/a2068402345783332864
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e215.38%2 of 13 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsJun 14: 0 · Jun 15: 1 · Jun 16: 2 · Jun 17: 0 · Jun 18: 1 · Jun 19: 1 · Jun 20: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc00160ea98>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc00160ea98>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00106a780>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-26pckg, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-26pckg" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00106a780>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-26pckg, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-26pckg" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068402345783332864
n/a2068399947908124672
Deserialization Error: no output from command
full failure pattern:
Deserialization Error: no output from command
Signal: NoiseSignal: Noise — source deserialization/no-output error — New failure pattern — no prior history
e2e215.38%2 of 13 job runs affectedNoiseSignal: Noise — source deserialization/no-output error — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 2none
Full failure examples (2)
go:2125 +0x47 fp=0xc000be4d28 sp=0xc000be4cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc00162c1e0)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc000be4d60 sp=0xc000be4d28 pc=0x3084943
bufio.(*Reader).Peek(0xc00162c1e0, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc000be4d80 sp=0xc000be4d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc001359d40)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc000be4fc8 sp=0xc000be4d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc000be4fe0 sp=0xc000be4fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000be4fe8 sp=0xc000be4fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 173
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 189 gp=0xc001001dc0 m=nil [select, 1 minutes]:
runtime.gopark(0xc0016b3f48?, 0x2?, 0xc0?, 0x3d?, 0xc0016b3ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0016b3d78 sp=0xc0016b3d58 pc=0x2f1e50e
runtime.selectgo(0xc0016b3f48, 0xc0016b3ef0, 0xc000c97240?, 0x0, 0xc0018e37d0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc0016b3eb8 sp=0xc0016b3d78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc001359d40)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc0016b3fc8 sp=0xc0016b3eb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc0016b3fe0 sp=0xc0016b3fc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0016b3fe8 sp=0xc0016b3fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 173
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 235 gp=0xc0016a4fc0 m=nil [select]:
runtime.gopark(0xc0013df768?, 0x2?, 0x90?, 0x1f?, 0xc0013df75c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0013df5e8 sp=0xc0013df5c8 pc=0x2f1e50e
runtime.selectgo(0xc0013df768, 0xc0013df758, 0xc000f5d760?, 0x0, 0x36ecd5c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc0013df728 sp=0xc0013df5e8 pc=0x2ef92f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x92b8658, 0xc0018e0b60}, 0x8704ddd?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc0013df798 sp=0xc0013df728 pc=0x36ec4b6
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x930a440, {0x92b8658, 0xc0018e0b60}, 0x2f1baf3)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc0013dfa98 sp=0xc0013df798 pc=0x3ded372
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x92b8658, 0xc0018e0b60}, 0xc001037308, {0xc0010125e8, 0x12}, {0xc001546108, 0x12})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:240 +0x365 fp=0xc0013dfc20 sp=0xc0013dfa98 pc=0x6c2ef65
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x92b8658?, 0xc0018e0b60?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x72 fp=0xc0013dfcc8 sp=0xc0013dfc20 pc=0x6c2e992
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x92b8658?, 0xc0018e0b60?}, 0xc0018e0b70?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc0013dfd20 sp=0xc0013dfcc8 pc=0x369fa82
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x92b8658, 0xc0018e0b60}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc0013dfe68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc0013dfd98 sp=0xc0013dfd20 pc=0x369db2e
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x92b85e8, 0xc0008bb540}, 0xc001037308, {0xc0010125e8, 0x12}, {0xc001546108, 0x12}, 0x274a48a7800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:196 +0x285 fp=0xc0013dfef0 sp=0xc0013dfd98 pc=0x6c2e705
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:522 +0x7e fp=0xc0013dff78 sp=0xc0013dfef0 pc=0x6c325fe
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc0013dffe0 sp=0xc0013dff78 pc=0x4a4be30
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0013dffe8 sp=0xc0013dffe0 pc=0x2f26cc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 214
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x2f28ae3
rdx    0x0
rdi    0xc90cc70
rsi    0x80
rbp    0x7ffcdfaaa7b0
rsp    0x7ffcdfaaa768
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1c
r13    0x1
r14    0xc90a6e0
r15    0x7
rip    0x2f28ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
go:2125 +0x47 fp=0xc000be4d28 sp=0xc000be4cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc00162c1e0)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc000be4d60 sp=0xc000be4d28 pc=0x3084943
bufio.(*Reader).Peek(0xc00162c1e0, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc000be4d80 sp=0xc000be4d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc001359d40)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc000be4fc8 sp=0xc000be4d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc000be4fe0 sp=0xc000be4fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000be4fe8 sp=0xc000be4fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 173
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 189 gp=0xc001001dc0 m=nil [select, 1 minutes]:
runtime.gopark(0xc0016b3f48?, 0x2?, 0xc0?, 0x3d?, 0xc0016b3ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0016b3d78 sp=0xc0016b3d58 pc=0x2f1e50e
runtime.selectgo(0xc0016b3f48, 0xc0016b3ef0, 0xc000c97240?, 0x0, 0xc0018e37d0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc0016b3eb8 sp=0xc0016b3d78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc001359d40)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc0016b3fc8 sp=0xc0016b3eb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc0016b3fe0 sp=0xc0016b3fc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0016b3fe8 sp=0xc0016b3fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 173
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 235 gp=0xc0016a4fc0 m=nil [select]:
runtime.gopark(0xc0013df768?, 0x2?, 0x90?, 0x1f?, 0xc0013df75c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc0013df5e8 sp=0xc0013df5c8 pc=0x2f1e50e
runtime.selectgo(0xc0013df768, 0xc0013df758, 0xc000f5d760?, 0x0, 0x36ecd5c?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc0013df728 sp=0xc0013df5e8 pc=0x2ef92f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x92b8658, 0xc0018e0b60}, 0x8704ddd?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc0013df798 sp=0xc0013df728 pc=0x36ec4b6
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x930a440, {0x92b8658, 0xc0018e0b60}, 0x2f1baf3)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc0013dfa98 sp=0xc0013df798 pc=0x3ded372
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x92b8658, 0xc0018e0b60}, 0xc001037308, {0xc0010125e8, 0x12}, {0xc001546108, 0x12})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:240 +0x365 fp=0xc0013dfc20 sp=0xc0013dfa98 pc=0x6c2ef65
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x92b8658?, 0xc0018e0b60?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:198 +0x72 fp=0xc0013dfcc8 sp=0xc0013dfc20 pc=0x6c2e992
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x92b8658?, 0xc0018e0b60?}, 0xc0018e0b70?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc0013dfd20 sp=0xc0013dfcc8 pc=0x369fa82
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x92b8658, 0xc0018e0b60}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc0013dfe68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc0013dfd98 sp=0xc0013dfd20 pc=0x369db2e
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x92b85e8, 0xc0008bb540}, 0xc001037308, {0xc0010125e8, 0x12}, {0xc001546108, 0x12}, 0x274a48a7800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:196 +0x285 fp=0xc0013dfef0 sp=0xc0013dfd98 pc=0x6c2e705
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/hcp_helper.go:522 +0x7e fp=0xc0013dff78 sp=0xc0013dfef0 pc=0x6c325fe
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:93 +0x50 fp=0xc0013dffe0 sp=0xc0013dff78 pc=0x4a4be30
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0013dffe8 sp=0xc0013dffe0 pc=0x2f26cc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 214
	/go/pkg/mod/golang.org/x/sync@v0.20.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x2f28ae3
rdx    0x0
rdi    0xc90cc70
rsi    0x80
rbp    0x7ffcdfaaa7b0
rsp    0x7ffcdfaaa768
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x1c
r13    0x1
r14    0xc90a6e0
r15    0x7
rip    0x2f28ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
53 fp=0xc0000c9d80 sp=0xc0000c9d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0010a6b40)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc0000c9fc8 sp=0xc0000c9d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc0000c9fe0 sp=0xc0000c9fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000c9fe8 sp=0xc0000c9fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 212
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 236 gp=0xc000e1ec40 m=nil [select, 2 minutes]:
runtime.gopark(0xc001303f48?, 0x2?, 0xc0?, 0x3d?, 0xc001303ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000382d78 sp=0xc000382d58 pc=0x2f1e50e
runtime.selectgo(0xc000382f48, 0xc001303ef0, 0xc0008e8b40?, 0x0, 0x0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000382eb8 sp=0xc000382d78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc0018445a0)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc000382fc8 sp=0xc000382eb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc000382fe0 sp=0xc000382fc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000382fe8 sp=0xc000382fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 197
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 235 gp=0xc000682e00 m=nil [IO wait, 2 minutes]:
runtime.gopark(0xc0000d5760?, 0xc001105788?, 0xc5?, 0xbe?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001105768 sp=0xc001105748 pc=0x2f1e50e
runtime.netpollblock(0x2f88ad8?, 0x2eaa066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc0011057a0 sp=0xc001105768 pc=0x2eddfb7
internal/poll.runtime_pollWait(0x7f51f5ba3e00, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc0011057c0 sp=0xc0011057a0 pc=0x2f1cda5
internal/poll.(*pollDesc).wait(0xc001866680?, 0xc001728a00?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0011057e8 sp=0xc0011057c0 pc=0x2fab907
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001866680, {0xc001728a00, 0x2500, 0x2500})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc001105880 sp=0xc0011057e8 pc=0x2facd39
net.(*netFD).Read(0xc001866680, {0xc001728a00?, 0xc001728a00?, 0x5?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc0011058c8 sp=0xc001105880 pc=0x31712e5
net.(*conn).Read(0xc000d6aa10, {0xc001728a00?, 0x7f51f42af700?, 0x7f523c789f30?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc001105910 sp=0xc0011058c8 pc=0x3183385
crypto/tls.(*atLeastReader).Read(0xc001800f60, {0xc001728a00?, 0x240a?, 0x0?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc001105958 sp=0xc001105910 pc=0x31eef1b
bytes.(*Buffer).ReadFrom(0xc000c129a8, {0x9220320, 0xc001800f60})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc0011059b0 sp=0xc001105958 pc=0x2fef578
crypto/tls.(*Conn).readFromUntil(0xc000c12708, {0x9221e00, 0xc000d6aa10}, 0xc001105a18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0011059e8 sp=0xc0011059b0 pc=0x31ef0fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000c12708, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc001105c60 sp=0xc0011059e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000c12708, {0xc001892000, 0x1000, 0xc0008673b0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc001105cc8 sp=0xc001105c60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc0018445a0, {0xc001892000?, 0x32dcb05?, 0x921acc0?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc001105d28 sp=0xc001105cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc000e9db60)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc001105d60 sp=0xc001105d28 pc=0x3084943
bufio.(*Reader).Peek(0xc000e9db60, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc001105d80 sp=0xc001105d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0018445a0)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc001105fc8 sp=0xc001105d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc001105fe0 sp=0xc001105fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001105fe8 sp=0xc001105fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 197
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

rax    0xca
rbx    0x0
rcx    0x2f28ae3
rdx    0x0
rdi    0xc90cc70
rsi    0x80
rbp    0x7fff4aa32180
rsp    0x7fff4aa32138
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7fff4aa32180
r13    0x1
r14    0xc90a6e0
r15    0x5
rip    0x2f28ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
53 fp=0xc0000c9d80 sp=0xc0000c9d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0010a6b40)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc0000c9fc8 sp=0xc0000c9d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc0000c9fe0 sp=0xc0000c9fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000c9fe8 sp=0xc0000c9fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 212
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

goroutine 236 gp=0xc000e1ec40 m=nil [select, 2 minutes]:
runtime.gopark(0xc001303f48?, 0x2?, 0xc0?, 0x3d?, 0xc001303ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000382d78 sp=0xc000382d58 pc=0x2f1e50e
runtime.selectgo(0xc000382f48, 0xc001303ef0, 0xc0008e8b40?, 0x0, 0x0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000382eb8 sp=0xc000382d78 pc=0x2ef92f7
net/http.(*persistConn).writeLoop(0xc0018445a0)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc000382fc8 sp=0xc000382eb8 pc=0x32e6ba7
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc000382fe0 sp=0xc000382fc8 pc=0x32e3845
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000382fe8 sp=0xc000382fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 197
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 235 gp=0xc000682e00 m=nil [IO wait, 2 minutes]:
runtime.gopark(0xc0000d5760?, 0xc001105788?, 0xc5?, 0xbe?, 0xb?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001105768 sp=0xc001105748 pc=0x2f1e50e
runtime.netpollblock(0x2f88ad8?, 0x2eaa066?, 0x0?)
	/usr/lib/golang/src/runtime/netpoll.go:575 +0xf7 fp=0xc0011057a0 sp=0xc001105768 pc=0x2eddfb7
internal/poll.runtime_pollWait(0x7f51f5ba3e00, 0x72)
	/usr/lib/golang/src/runtime/netpoll.go:351 +0x85 fp=0xc0011057c0 sp=0xc0011057a0 pc=0x2f1cda5
internal/poll.(*pollDesc).wait(0xc001866680?, 0xc001728a00?, 0x0)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:84 +0x27 fp=0xc0011057e8 sp=0xc0011057c0 pc=0x2fab907
internal/poll.(*pollDesc).waitRead(...)
	/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc001866680, {0xc001728a00, 0x2500, 0x2500})
	/usr/lib/golang/src/internal/poll/fd_unix.go:165 +0x279 fp=0xc001105880 sp=0xc0011057e8 pc=0x2facd39
net.(*netFD).Read(0xc001866680, {0xc001728a00?, 0xc001728a00?, 0x5?})
	/usr/lib/golang/src/net/fd_posix.go:68 +0x25 fp=0xc0011058c8 sp=0xc001105880 pc=0x31712e5
net.(*conn).Read(0xc000d6aa10, {0xc001728a00?, 0x7f51f42af700?, 0x7f523c789f30?})
	/usr/lib/golang/src/net/net.go:196 +0x45 fp=0xc001105910 sp=0xc0011058c8 pc=0x3183385
crypto/tls.(*atLeastReader).Read(0xc001800f60, {0xc001728a00?, 0x240a?, 0x0?})
	/usr/lib/golang/src/crypto/tls/conn.go:819 +0x3b fp=0xc001105958 sp=0xc001105910 pc=0x31eef1b
bytes.(*Buffer).ReadFrom(0xc000c129a8, {0x9220320, 0xc001800f60})
	/usr/lib/golang/src/bytes/buffer.go:217 +0x98 fp=0xc0011059b0 sp=0xc001105958 pc=0x2fef578
crypto/tls.(*Conn).readFromUntil(0xc000c12708, {0x9221e00, 0xc000d6aa10}, 0xc001105a18?)
	/usr/lib/golang/src/crypto/tls/conn.go:841 +0xde fp=0xc0011059e8 sp=0xc0011059b0 pc=0x31ef0fe
crypto/tls.(*Conn).readRecordOrCCS(0xc000c12708, 0x0)
	/usr/lib/golang/src/crypto/tls/conn.go:630 +0x3db fp=0xc001105c60 sp=0xc0011059e8 pc=0x31ec17b
crypto/tls.(*Conn).readRecord(...)
	/usr/lib/golang/src/crypto/tls/conn.go:592
crypto/tls.(*Conn).Read(0xc000c12708, {0xc001892000, 0x1000, 0xc0008673b0?})
	/usr/lib/golang/src/crypto/tls/conn.go:1397 +0x145 fp=0xc001105cc8 sp=0xc001105c60 pc=0x31f2a85
net/http.(*persistConn).Read(0xc0018445a0, {0xc001892000?, 0x32dcb05?, 0x921acc0?})
	/usr/lib/golang/src/net/http/transport.go:2125 +0x47 fp=0xc001105d28 sp=0xc001105cc8 pc=0x32e4407
bufio.(*Reader).fill(0xc000e9db60)
	/usr/lib/golang/src/bufio/bufio.go:113 +0x103 fp=0xc001105d60 sp=0xc001105d28 pc=0x3084943
bufio.(*Reader).Peek(0xc000e9db60, 0x1)
	/usr/lib/golang/src/bufio/bufio.go:152 +0x53 fp=0xc001105d80 sp=0xc001105d60 pc=0x3084ad3
net/http.(*persistConn).readLoop(0xc0018445a0)
	/usr/lib/golang/src/net/http/transport.go:2278 +0x172 fp=0xc001105fc8 sp=0xc001105d80 pc=0x32e4f12
net/http.(*Transport).dialConn.gowrap2()
	/usr/lib/golang/src/net/http/transport.go:1947 +0x25 fp=0xc001105fe0 sp=0xc001105fc8 pc=0x32e38a5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001105fe8 sp=0xc001105fe0 pc=0x2f26cc1
created by net/http.(*Transport).dialConn in goroutine 197
	/usr/lib/golang/src/net/http/transport.go:1947 +0x174f

rax    0xca
rbx    0x0
rcx    0x2f28ae3
rdx    0x0
rdi    0xc90cc70
rsi    0x80
rbp    0x7fff4aa32180
rsp    0x7fff4aa32138
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x7fff4aa32180
r13    0x1
r14    0xc90a6e0
r15    0x5
rip    0x2f28ae1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068465010476060672
n/a2068402345783332864
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e646.15%6 of 13 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 8none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-55jt6c" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00146eb10>: 
    failed to create HCP cluster cluster-zstream-4-22-55jt6c, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-55jt6c in resource group rg-zstream-upgrade-4-22-gmxf8f, error: failed waiting for cluster="cluster-zstream-4-22-55jt6c" in resourcegroup="rg-zstream-upgrade-4-22-gmxf8f" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-55jt6c in resource group rg-zstream-upgrade-4-22-gmxf8f, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-55jt6c" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc00146eb10>: 
    failed to create HCP cluster cluster-zstream-4-22-55jt6c, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-55jt6c in resource group rg-zstream-upgrade-4-22-gmxf8f, error: failed waiting for cluster="cluster-zstream-4-22-55jt6c" in resourcegroup="rg-zstream-upgrade-4-22-gmxf8f" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-55jt6c in resource group rg-zstream-upgrade-4-22-gmxf8f, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-5-0-lvjsdl/cluster-nightly-5-0-hxzbgf should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0016204e0>: 
    failed to create HCP cluster cluster-nightly-5-0-hxzbgf, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-hxzbgf in resource group rg-nightly-5-0-lvjsdl, error: failed waiting for cluster="cluster-nightly-5-0-hxzbgf" in resourcegroup="rg-nightly-5-0-lvjsdl" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-hxzbgf in resource group rg-nightly-5-0-lvjsdl, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-5-0-lvjsdl/cluster-nightly-5-0-hxzbgf should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0016204e0>: 
    failed to create HCP cluster cluster-nightly-5-0-hxzbgf, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-hxzbgf in resource group rg-nightly-5-0-lvjsdl, error: failed waiting for cluster="cluster-nightly-5-0-hxzbgf" in resourcegroup="rg-nightly-5-0-lvjsdl" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-hxzbgf in resource group rg-nightly-5-0-lvjsdl, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-22-lklm69/cluster-candidate-4-22-f4gmfq should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0014d4690>: 
    failed to create HCP cluster cluster-candidate-4-22-f4gmfq, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-f4gmfq in resource group rg-candidate-4-22-lklm69, error: failed waiting for cluster="cluster-candidate-4-22-f4gmfq" in resourcegroup="rg-candidate-4-22-lklm69" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-f4gmfq in resource group rg-candidate-4-22-lklm69, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-22-lklm69/cluster-candidate-4-22-f4gmfq should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0014d4690>: 
    failed to create HCP cluster cluster-candidate-4-22-f4gmfq, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-f4gmfq in resource group rg-candidate-4-22-lklm69, error: failed waiting for cluster="cluster-candidate-4-22-f4gmfq" in resourcegroup="rg-candidate-4-22-lklm69" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-22-f4gmfq in resource group rg-candidate-4-22-lklm69, error: context deadline exceeded
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.223
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.222
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.01
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
Affected runs (6)
Date (UTC)Associated PRProw job
n/a2068469052392607744
n/a2068465010476060672
n/a2068443878926585856
n/a2068402345783332864
n/a2068216057461477376
n/a2068215937068175360
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
full failure pattern:
route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e215.38%2 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 10 · Jun 20: 7none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: failed to verify simple web app on back-level cluster version 4.19
Unexpected error:
    <*fmt.wrapError | 0xc000f540e0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-zgg49.apps.aro.h8q7w4u9x7c7g6v.x1ie.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-zgg49.apps.aro.h8q7w4u9x7c7g6v.x1ie.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:194]: failed to verify simple web app on back-level cluster version 4.19
Unexpected error:
    <*fmt.wrapError | 0xc000f540e0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-zgg49.apps.aro.h8q7w4u9x7c7g6v.x1ie.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-zgg49.apps.aro.h8q7w4u9x7c7g6v.x1ie.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-wxppcl"
Unexpected error:
    <*fmt.wrapError | 0xc00012ae20>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-4d2pm.apps.aro.f2j5d7e4q4x3n6o.g26p.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-4d2pm.apps.aro.f2j5d7e4q4x3n6o.g26p.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-wxppcl"
Unexpected error:
    <*fmt.wrapError | 0xc00012ae20>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-4d2pm.apps.aro.f2j5d7e4q4x3n6o.g26p.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-4d2pm.apps.aro.f2j5d7e4q4x3n6o.g26p.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-6z4flh"
Unexpected error:
    <*fmt.wrapError | 0xc000954aa0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-dg7r4.apps.aro.k4f9f7h5q5s9h1o.o4cb.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-dg7r4.apps.aro.k4f9f7h5q5s9h1o.o4cb.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-6z4flh"
Unexpected error:
    <*fmt.wrapError | 0xc000954aa0>: 
    route was never reachable: Get "https://agnhost-e2e-serving-app-dg7r4.apps.aro.k4f9f7h5q5s9h1o.o4cb.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-dg7r4.apps.aro.k4f9f7h5q5s9h1o.o4cb.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.212
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.192
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068465010476060672
n/a2068402345783332864
VerifySimpleWebApp failed: route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
full failure pattern:
VerifySimpleWebApp failed: route was never reachable: Get "<url>": dial tcp: lookup <host> on <dns-server>: no such host
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e215.38%2 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 2 · Jun 20: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0021b6000>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-8872n.apps.aro.cilium-cluster.moy3.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-8872n.apps.aro.cilium-cluster.moy3.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0021b6000>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-8872n.apps.aro.cilium-cluster.moy3.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-8872n.apps.aro.cilium-cluster.moy3.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0012542a0>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-j5bdd.apps.aro.cilium-cl.17y9.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-j5bdd.apps.aro.cilium-cl.17y9.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0012542a0>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-j5bdd.apps.aro.cilium-cl.17y9.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-j5bdd.apps.aro.cilium-cl.17y9.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0011fe378>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-6tkcv.apps.aro.cilium-cluster.aq51.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-6tkcv.apps.aro.cilium-cluster.aq51.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc0011fe378>: 
    VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-6tkcv.apps.aro.cilium-cluster.aq51.brazilsouth.aroapp-hcp.io": dial tcp: lookup agnhost-e2e-serving-app-6tkcv.apps.aro.cilium-cluster.aq51.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068465010476060672
n/a2068402345783332864
dial tcp: lookup console-openshift-console
full failure pattern:
dial tcp: lookup console-openshift-console
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e215.38%2 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.003s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc0008c56d0>: 
    dial tcp: lookup console-openshift-console.apps.aro.f4y0j8z4x0b5v2h.j4r9.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.003s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc0008c56d0>: 
    dial tcp: lookup console-openshift-console.apps.aro.f4y0j8z4x0b5v2h.j4r9.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.005s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc000a493b0>: 
    dial tcp: lookup console-openshift-console.apps.aro.s3s6l6j1t7z6u0i.vxvo.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:167]: Timed out after 600.005s.
expect ingress certificate to be signed by a trusted Azure CA
Expected success, but got an error:
    <*net.OpError | 0xc000a493b0>: 
    dial tcp: lookup console-openshift-console.apps.aro.s3s6l6j1t7z6u0i.vxvo.brazilsouth.aroapp-hcp.io on 172.30.0.10:53: no such host
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2068465010476060672
n/a2068402345783332864
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Pro...
full failure pattern:
cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e17.69%1 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc0013bc3f0>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:244]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc0013bc3f0>: 
    context deadline exceeded
    cluster operators not available: console (Degraded=True, Progressing=False, Available=False, Upgradeable=True, EvaluationConditionsDetected=Unknown), ingress (Available=False, Progressing=True, Degraded=True, Upgradeable=True, EvaluationConditionsDetected=False)
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068465010476060672
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e17.69%1 of 13 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:126]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000d52e10>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-5c9rf9, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5c9rf9" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:126]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapErrors | 0xc000d52e10>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group autoscaling-cluster-5c9rf9, error: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-5c9rf9" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068402345783332864
add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
full failure pattern:
add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e17.69%1 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:132]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc0007bf3b0>: 
    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:132]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc0007bf3b0>: 
    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068216024569745408
couldn't get current server API group list: Get "<url>": dial tcp <ip>:<port>: i/o timeout
full failure pattern:
couldn't get current server API group list: Get "<url>": dial tcp <ip>:<port>: i/o timeout
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e17.69%1 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 605.076s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:403 with:
kubectl get clusteroperators should succeed from authorized VM
Unexpected error:
    <*errors.errorString | 0xc000a7a390>: 
    E0621 00:06:17.087501    3391 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.h5h5i6l8r7c3p7a.u2v6.brazilsouth.aroapp-hcp.io:443/api?timeout=32s\": dial tcp 74.163.139.192:443: i/o timeout"
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:407]: Timed out after 605.076s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:403 with:
kubectl get clusteroperators should succeed from authorized VM
Unexpected error:
    <*errors.errorString | 0xc000a7a390>: 
    E0621 00:06:17.087501    3391 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://api.h5h5i6l8r7c3p7a.u2v6.brazilsouth.aroapp-hcp.io:443/api?timeout=32s\": dial tcp 74.163.139.192:443: i/o timeout"
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068465010476060672
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...
full failure pattern:
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e17.69%1 of 13 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000cf4240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-t2dlwt" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000cf4240>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-t2dlwt" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelUpdate HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068465010476060672
failed to create namespace: Post "<url>": dial tcp <ip>:<port>: i/o timeout
full failure pattern:
failed to create namespace: Post "<url>": dial tcp <ip>:<port>: i/o timeout
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e17.69%1 of 13 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsJun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-qqg5xh"
Unexpected error:
    <*fmt.wrapError | 0xc000b1a060>: 
    failed to create namespace: Post "https://api.q8q3f7q1h7j5r8s.w6pe.brazilsouth.aroapp-hcp.io:443/api/v1/namespaces": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-qqg5xh"
Unexpected error:
    <*fmt.wrapError | 0xc000b1a060>: 
    failed to create namespace: Post "https://api.q8q3f7q1h7j5r8s.w6pe.brazilsouth.aroapp-hcp.io:443/api/v1/namespaces": dial tcp 74.163.139.192:443: i/o timeout
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2068402345783332864