No failure patterns match your search.
Environment: PROD
Failure patterns: 16 · Failures matched: 178 · Total runs: 60 · Runs affected: 26
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Deserialization Error: no output from commandfull failure pattern: Deserialization Error: no output from command Signal: NoiseSignal: Noise — source deserialization/no-output error | e2e | 1 | 1.67%1 of 60 job runs affected | NoiseSignal: Noise — source deserialization/no-output error | Show trend detailsJul 19: 0 · Jul 20: 2 · Jul 21: 0 · Jul 22: 0 · Jul 23: 0 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)053c60, 0xc0004e9b00}, {{0xc000f8eec0, 0x1, 0x1}, 0x34630b8a000, {0x871a24d, 0x8}, {{0x0, ...}, ...}})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x252 fp=0xc00190f730 sp=0xc00190f618 pc=0x6c9d292
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000bc26e0, {0x7f2434053c60, 0xc0004e9b00})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262 +0x250 fp=0xc00190f888 sp=0xc00190f730 pc=0x6c9c9f0
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f2434053c60?, 0xc0004e9b00?})
<autogenerated>:1 +0x33 fp=0xc00190f8b0 sp=0xc00190f888 pc=0x6cbcad3
runtime.call16(0xc0004bf380, 0xc0006f95c0, 0x0, 0x0, 0x0, 0x10, 0xc00190fb08)
/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc00190f8d0 sp=0xc00190f8b0 pc=0x2f2a223
runtime.reflectcall(0x80e5200?, 0xc0004e9b00?, 0x2?, 0x8751e1d?, 0x0?, 0x12?, 0x7c9d5e0?)
<autogenerated>:1 +0x36 fp=0xc00190f910 sp=0xc00190f8d0 pc=0x2f32ab6
reflect.Value.call({0x75465a0?, 0xc0006f95c0?, 0xc000d70d20?}, {0x870dbd9, 0x4}, {0xc000c24f60, 0x1, 0x0?})
/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc00190fe70 sp=0xc00190f910 pc=0x2f65c66
reflect.Value.Call({0x75465a0?, 0xc0006f95c0?, 0xc000d70d20?}, {0xc000c24f60?, 0xc000bc26e0?, 0x932c968?})
/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc00190fee8 sp=0xc00190fe70 pc=0x2f64d59
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x93472c8?, 0xc0004e9b00?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc00190ff88 sp=0xc00190fee8 pc=0x3388c70
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc00190ffe0 sp=0xc00190ff88 pc=0x339b73b
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00190ffe8 sp=0xc00190ffe0 pc=0x2f2bcc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 120
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
goroutine 377 gp=0xc0016fa1c0 m=nil [select]:
runtime.gopark(0xc000d5d900?, 0x2?, 0x0?, 0xbe?, 0xc000d5d8f4?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000d5d780 sp=0xc000d5d760 pc=0x2f2350e
runtime.selectgo(0xc000d5d900, 0xc000d5d8f0, 0xc0015dec80?, 0x0, 0x36f17dc?, 0x1)
/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000d5d8c0 sp=0xc000d5d780 pc=0x2efe2f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x932c9d8, 0xc00071cbd0}, 0x8775b21?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc000d5d930 sp=0xc000d5d8c0 pc=0x36f0f36
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x9377d20, {0x932c9d8, 0xc00071cbd0}, 0x11)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc000d5dc30 sp=0xc000d5d930 pc=0x3dfc6f2
github.com/Azure/ARO-HCP/test/util/framework.DeleteResourceGroup({0x7f2434053c60, 0xc0004e9b00}, 0xc00154a960, 0xc000cc2de0, {0xc000c6ce10, 0x11}, 0x0, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/resourcegroup_helper.go:172 +0x6aa fp=0xc000d5dd90 sp=0xc000d5dc30 pc=0x6ca92ca
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000bc26e0, {0x7f2434053c60, 0xc0004e9b00}, {0xc000c6ce10, 0x11}, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:568 +0x8a5 fp=0xc000d5df08 sp=0xc000d5dd90 pc=0x6c9fd05
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7f2434053c60?, 0xc0004e9b00?})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:360 +0x151 fp=0xc000d5dfc0 sp=0xc000d5df08 pc=0x6c9d791
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:369 +0x28 fp=0xc000d5dfe0 sp=0xc000d5dfc0 pc=0x6c9d608
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d5dfe8 sp=0xc000d5dfe0 pc=0x2f2bcc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 420
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353 +0x98
rax 0xca
rbx 0x0
rcx 0x2f2dae3
rdx 0x0
rdi 0xc9599f0
rsi 0x80
rbp 0x7ffc3cc57520
rsp 0x7ffc3cc574d8
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x7
r13 0x1
r14 0xc957640
r15 0x7
rip 0x2f2dae1
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserialization Error: no output from command
053c60, 0xc0004e9b00}, {{0xc000f8eec0, 0x1, 0x1}, 0x34630b8a000, {0x871a24d, 0x8}, {{0x0, ...}, ...}})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x252 fp=0xc00190f730 sp=0xc00190f618 pc=0x6c9d292
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000bc26e0, {0x7f2434053c60, 0xc0004e9b00})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262 +0x250 fp=0xc00190f888 sp=0xc00190f730 pc=0x6c9c9f0
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f2434053c60?, 0xc0004e9b00?})
<autogenerated>:1 +0x33 fp=0xc00190f8b0 sp=0xc00190f888 pc=0x6cbcad3
runtime.call16(0xc0004bf380, 0xc0006f95c0, 0x0, 0x0, 0x0, 0x10, 0xc00190fb08)
/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc00190f8d0 sp=0xc00190f8b0 pc=0x2f2a223
runtime.reflectcall(0x80e5200?, 0xc0004e9b00?, 0x2?, 0x8751e1d?, 0x0?, 0x12?, 0x7c9d5e0?)
<autogenerated>:1 +0x36 fp=0xc00190f910 sp=0xc00190f8d0 pc=0x2f32ab6
reflect.Value.call({0x75465a0?, 0xc0006f95c0?, 0xc000d70d20?}, {0x870dbd9, 0x4}, {0xc000c24f60, 0x1, 0x0?})
/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc00190fe70 sp=0xc00190f910 pc=0x2f65c66
reflect.Value.Call({0x75465a0?, 0xc0006f95c0?, 0xc000d70d20?}, {0xc000c24f60?, 0xc000bc26e0?, 0x932c968?})
/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc00190fee8 sp=0xc00190fe70 pc=0x2f64d59
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x93472c8?, 0xc0004e9b00?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc00190ff88 sp=0xc00190fee8 pc=0x3388c70
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc00190ffe0 sp=0xc00190ff88 pc=0x339b73b
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc00190ffe8 sp=0xc00190ffe0 pc=0x2f2bcc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 120
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
goroutine 377 gp=0xc0016fa1c0 m=nil [select]:
runtime.gopark(0xc000d5d900?, 0x2?, 0x0?, 0xbe?, 0xc000d5d8f4?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000d5d780 sp=0xc000d5d760 pc=0x2f2350e
runtime.selectgo(0xc000d5d900, 0xc000d5d8f0, 0xc0015dec80?, 0x0, 0x36f17dc?, 0x1)
/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000d5d8c0 sp=0xc000d5d780 pc=0x2efe2f7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x932c9d8, 0xc00071cbd0}, 0x8775b21?)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/internal/shared/shared.go:36 +0x76 fp=0xc000d5d930 sp=0xc000d5d8c0 pc=0x36f0f36
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x9377d20, {0x932c9d8, 0xc00071cbd0}, 0x11)
/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.21.1/runtime/poller.go:284 +0x732 fp=0xc000d5dc30 sp=0xc000d5d930 pc=0x3dfc6f2
github.com/Azure/ARO-HCP/test/util/framework.DeleteResourceGroup({0x7f2434053c60, 0xc0004e9b00}, 0xc00154a960, 0xc000cc2de0, {0xc000c6ce10, 0x11}, 0x0, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/resourcegroup_helper.go:172 +0x6aa fp=0xc000d5dd90 sp=0xc000d5dc30 pc=0x6ca92ca
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).cleanupResourceGroup(0xc000bc26e0, {0x7f2434053c60, 0xc0004e9b00}, {0xc000c6ce10, 0x11}, 0x34630b8a000)
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:568 +0x8a5 fp=0xc000d5df08 sp=0xc000d5dd90 pc=0x6c9fd05
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.func1({0x7f2434053c60?, 0xc0004e9b00?})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:360 +0x151 fp=0xc000d5dfc0 sp=0xc000d5df08 pc=0x6c9d791
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups.gowrap1()
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:369 +0x28 fp=0xc000d5dfe0 sp=0xc000d5dfc0 pc=0x6c9d608
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000d5dfe8 sp=0xc000d5dfe0 pc=0x2f2bcc1
created by github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups in goroutine 420
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:353 +0x98
rax 0xca
rbx 0x0
rcx 0x2f2dae3
rdx 0x0
rdi 0xc9599f0
rsi 0x80
rbp 0x7ffc3cc57520
rsp 0x7ffc3cc574d8
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x7
r13 0x1
r14 0xc957640
r15 0x7
rip 0x2f2dae1
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserialization Error: no output from command3089fb7
io.ReadAtLeast({0x92938e0, 0xc0014ab680}, {0xc0014b4204, 0x9, 0x9}, 0x9)
/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000af7d00 sp=0xc000af7cb8 pc=0x2f7e06e
io.ReadFull(...)
/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0014b4204, 0x9, 0xc000000063?}, {0x92938e0?, 0xc0014ab680?})
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:250 +0x65 fp=0xc000af7d50 sp=0xc000af7d00 pc=0x35ee225
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0014b41c0)
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:513 +0x6b fp=0xc000af7dd8 sp=0xc000af7d50 pc=0x35eedeb
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0014b41c0)
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:572 +0x18 fp=0xc000af7e18 sp=0xc000af7dd8 pc=0x35ef558
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000af7fa8)
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:1878 +0xca fp=0xc000af7f60 sp=0xc000af7e18 pc=0x360154a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0016c0a80)
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:1760 +0x52 fp=0xc000af7fc8 sp=0xc000af7f60 pc=0x3600a72
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:554 +0x25 fp=0xc000af7fe0 sp=0xc000af7fc8 pc=0x35fa1a5
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000af7fe8 sp=0xc000af7fe0 pc=0x2f2bcc1
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 134
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:554 +0xda5
goroutine 316 gp=0xc000697dc0 m=nil [sync.WaitGroup.Wait]:
runtime.gopark(0xc0016115b0?, 0x2eba874?, 0x20?, 0x99?, 0x200c001546540?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001611550 sp=0xc001611530 pc=0x2f2350e
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc001558348, 0x0, 0x1, 0x0, 0x19)
/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc0016115b8 sp=0xc001611550 pc=0x2eff59d
sync.runtime_SemacquireWaitGroup(0xc000d25920?, 0xc0?)
/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc0016115f0 sp=0xc0016115b8 pc=0x2f24c2e
sync.(*WaitGroup).Wait(0xc001558340)
/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc001611618 sp=0xc0016115f0 pc=0x2f48405
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc000eba000, {0x7f6c39063a60, 0xc0015a32c0}, {{0xc000ec09c0, 0x1, 0x1}, 0x34630b8a000, {0x871a24d, 0x8}, {{0x0, ...}, ...}})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x252 fp=0xc001611730 sp=0xc001611618 pc=0x6c9d292
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000eba000, {0x7f6c39063a60, 0xc0015a32c0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262 +0x250 fp=0xc001611888 sp=0xc001611730 pc=0x6c9c9f0
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f6c39063a60?, 0xc0015a32c0?})
<autogenerated>:1 +0x33 fp=0xc0016118b0 sp=0xc001611888 pc=0x6cbcad3
runtime.call16(0xc000e94300, 0xc000ae0410, 0x0, 0x0, 0x0, 0x10, 0xc001611b08)
/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc0016118d0 sp=0xc0016118b0 pc=0x2f2a223
runtime.reflectcall(0x80e5200?, 0xc0015a32c0?, 0x2?, 0x8751e1d?, 0x0?, 0x12?, 0x7c9d5e0?)
<autogenerated>:1 +0x36 fp=0xc001611910 sp=0xc0016118d0 pc=0x2f32ab6
reflect.Value.call({0x75465a0?, 0xc000ae0410?, 0xc000e8c780?}, {0x870dbd9, 0x4}, {0xc000879f60, 0x1, 0xc001476a00?})
/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc001611e70 sp=0xc001611910 pc=0x2f65c66
reflect.Value.Call({0x75465a0?, 0xc000ae0410?, 0xc000879f68?}, {0xc000879f60?, 0xc000eba000?, 0x932c968?})
/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc001611ee8 sp=0xc001611e70 pc=0x2f64d59
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x93472c8?, 0xc0015a32c0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc001611f88 sp=0xc001611ee8 pc=0x3388c70
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc001611fe0 sp=0xc001611f88 pc=0x339b73b
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001611fe8 sp=0xc001611fe0 pc=0x2f2bcc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 83
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
rax 0xca
rbx 0x0
rcx 0x2f2dae3
rdx 0x0
rdi 0xc9599f0
rsi 0x80
rbp 0x7ffe78eb87a8
rsp 0x7ffe78eb8760
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x4c
r13 0x1
r14 0xc957640
r15 0x1
rip 0x2f2dae1
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserialization Error: no output from command
3089fb7
io.ReadAtLeast({0x92938e0, 0xc0014ab680}, {0xc0014b4204, 0x9, 0x9}, 0x9)
/usr/lib/golang/src/io/io.go:335 +0x8e fp=0xc000af7d00 sp=0xc000af7cb8 pc=0x2f7e06e
io.ReadFull(...)
/usr/lib/golang/src/io/io.go:354
golang.org/x/net/http2.readFrameHeader({0xc0014b4204, 0x9, 0xc000000063?}, {0x92938e0?, 0xc0014ab680?})
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:250 +0x65 fp=0xc000af7d50 sp=0xc000af7d00 pc=0x35ee225
golang.org/x/net/http2.(*Framer).ReadFrameHeader(0xc0014b41c0)
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:513 +0x6b fp=0xc000af7dd8 sp=0xc000af7d50 pc=0x35eedeb
golang.org/x/net/http2.(*Framer).ReadFrame(0xc0014b41c0)
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/frame.go:572 +0x18 fp=0xc000af7e18 sp=0xc000af7dd8 pc=0x35ef558
golang.org/x/net/http2.(*clientConnReadLoop).run(0xc000af7fa8)
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:1878 +0xca fp=0xc000af7f60 sp=0xc000af7e18 pc=0x360154a
golang.org/x/net/http2.(*ClientConn).readLoop(0xc0016c0a80)
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:1760 +0x52 fp=0xc000af7fc8 sp=0xc000af7f60 pc=0x3600a72
golang.org/x/net/http2.(*Transport).newClientConn.gowrap1()
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:554 +0x25 fp=0xc000af7fe0 sp=0xc000af7fc8 pc=0x35fa1a5
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000af7fe8 sp=0xc000af7fe0 pc=0x2f2bcc1
created by golang.org/x/net/http2.(*Transport).newClientConn in goroutine 134
/go/pkg/mod/golang.org/x/net@v0.55.0/http2/transport.go:554 +0xda5
goroutine 316 gp=0xc000697dc0 m=nil [sync.WaitGroup.Wait]:
runtime.gopark(0xc0016115b0?, 0x2eba874?, 0x20?, 0x99?, 0x200c001546540?)
/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001611550 sp=0xc001611530 pc=0x2f2350e
runtime.goparkunlock(...)
/usr/lib/golang/src/runtime/proc.go:466
runtime.semacquire1(0xc001558348, 0x0, 0x1, 0x0, 0x19)
/usr/lib/golang/src/runtime/sema.go:192 +0x21d fp=0xc0016115b8 sp=0xc001611550 pc=0x2eff59d
sync.runtime_SemacquireWaitGroup(0xc000d25920?, 0xc0?)
/usr/lib/golang/src/runtime/sema.go:114 +0x2e fp=0xc0016115f0 sp=0xc0016115b8 pc=0x2f24c2e
sync.(*WaitGroup).Wait(0xc001558340)
/usr/lib/golang/src/sync/waitgroup.go:206 +0x85 fp=0xc001611618 sp=0xc0016115f0 pc=0x2f48405
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).CleanupResourceGroups(0xc000eba000, {0x7f6c39063a60, 0xc0015a32c0}, {{0xc000ec09c0, 0x1, 0x1}, 0x34630b8a000, {0x871a24d, 0x8}, {{0x0, ...}, ...}})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:371 +0x252 fp=0xc001611730 sp=0xc001611618 pc=0x6c9d292
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources(0xc000eba000, {0x7f6c39063a60, 0xc0015a32c0})
/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262 +0x250 fp=0xc001611888 sp=0xc001611730 pc=0x6c9c9f0
github.com/Azure/ARO-HCP/test/util/framework.(*perItOrDescribeTestContext).deleteCreatedResources-fm({0x7f6c39063a60?, 0xc0015a32c0?})
<autogenerated>:1 +0x33 fp=0xc0016118b0 sp=0xc001611888 pc=0x6cbcad3
runtime.call16(0xc000e94300, 0xc000ae0410, 0x0, 0x0, 0x0, 0x10, 0xc001611b08)
/usr/lib/golang/src/runtime/asm_amd64.s:774 +0x43 fp=0xc0016118d0 sp=0xc0016118b0 pc=0x2f2a223
runtime.reflectcall(0x80e5200?, 0xc0015a32c0?, 0x2?, 0x8751e1d?, 0x0?, 0x12?, 0x7c9d5e0?)
<autogenerated>:1 +0x36 fp=0xc001611910 sp=0xc0016118d0 pc=0x2f32ab6
reflect.Value.call({0x75465a0?, 0xc000ae0410?, 0xc000e8c780?}, {0x870dbd9, 0x4}, {0xc000879f60, 0x1, 0xc001476a00?})
/usr/lib/golang/src/reflect/value.go:581 +0xcc6 fp=0xc001611e70 sp=0xc001611910 pc=0x2f65c66
reflect.Value.Call({0x75465a0?, 0xc000ae0410?, 0xc000879f68?}, {0xc000879f60?, 0xc000eba000?, 0x932c968?})
/usr/lib/golang/src/reflect/value.go:365 +0xb9 fp=0xc001611ee8 sp=0xc001611e70 pc=0x2f64d59
github.com/onsi/ginkgo/v2/internal.NewCleanupNode.func2({0x93472c8?, 0xc0015a32c0?})
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:589 +0x150 fp=0xc001611f88 sp=0xc001611ee8 pc=0x3388c70
github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 +0x7b fp=0xc001611fe0 sp=0xc001611f88 pc=0x339b73b
runtime.goexit({})
/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001611fe8 sp=0xc001611fe0 pc=0x2f2bcc1
created by github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 83
/go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 +0xd50
rax 0xca
rbx 0x0
rcx 0x2f2dae3
rdx 0x0
rdi 0xc9599f0
rsi 0x80
rbp 0x7ffe78eb87a8
rsp 0x7ffe78eb8760
r8 0x0
r9 0x0
r10 0x0
r11 0x286
r12 0x4c
r13 0x1
r14 0xc957640
r15 0x1
rip 0x2f2dae1
rflags 0x286
cs 0x33
fs 0x0
gs 0x0
Command Error: exit status 2
Deserialization Error: no output from commandContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
cluster entered terminal Failed state after role assignment deploymentfull failure pattern: cluster entered terminal Failed state after role assignment deployment Signal: IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | e2e | 9 | 15.00%9 of 60 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 4 day(s) — New failure pattern — no prior history | Show trend detailsJul 19: 0 · Jul 20: 1 · Jul 21: 2 · Jul 22: 4 · Jul 23: 2 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
<armredhatopenshifthcp.ProvisioningState>: ...
not to equal
<armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
<armredhatopenshifthcp.ProvisioningState>: ...
not to equal
<armredhatopenshifthcp.ProvisioningState>: ...fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
<armredhatopenshifthcp.ProvisioningState>: ...
not to equal
<armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
<armredhatopenshifthcp.ProvisioningState>: ...
not to equal
<armredhatopenshifthcp.ProvisioningState>: ...fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
<armredhatopenshifthcp.ProvisioningState>: ...
not to equal
<armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
<armredhatopenshifthcp.ProvisioningState>: ...
not to equal
<armredhatopenshifthcp.ProvisioningState>: ...Contributing tests (2)
Affected runs (9)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | e2e | 7 | 11.67%7 of 60 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s) | Show trend detailsJul 19: 1 · Jul 20: 1 · Jul 21: 0 · Jul 22: 3 · Jul 23: 4 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-hdmwqd" for cluster "cluster-candidate-4-20-hdmwqd"
Unexpected error:
<*fmt.wrapError | 0xc001062200>:
failed to create NodePool np-hdmwqd: failed waiting for nodepool="np-hdmwqd" for cluster "cluster-candidate-4-20-hdmwqd" in resourcegroup="rg-candidate-4-20-grr9djf6qtvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c91f066e-3821-44ee-bece-edbab2bb31db
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c91f066e-3821-44ee-bece-edbab2bb31db",
"name": "c91f066e-3821-44ee-bece-edbab2bb31db",
"status": "Failed",
"startTime": "2026-07-23T08:27:34.915224416Z",
"endTime": "2026-07-23T08:46:40.245736142Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-hdmwqd" for cluster "cluster-candidate-4-20-hdmwqd"
Unexpected error:
<*fmt.wrapError | 0xc001062200>:
failed to create NodePool np-hdmwqd: failed waiting for nodepool="np-hdmwqd" for cluster "cluster-candidate-4-20-hdmwqd" in resourcegroup="rg-candidate-4-20-grr9djf6qtvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c91f066e-3821-44ee-bece-edbab2bb31db
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c91f066e-3821-44ee-bece-edbab2bb31db",
"name": "c91f066e-3821-44ee-bece-edbab2bb31db",
"status": "Failed",
"startTime": "2026-07-23T08:27:34.915224416Z",
"endTime": "2026-07-23T08:46:40.245736142Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapError | 0xc00283c700>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-5676c2nc84jf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/81d6e351-7e5e-49b4-85a5-302b276cc8f6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/81d6e351-7e5e-49b4-85a5-302b276cc8f6",
"name": "81d6e351-7e5e-49b4-85a5-302b276cc8f6",
"status": "Failed",
"startTime": "2026-07-23T07:46:53.376501029Z",
"endTime": "2026-07-23T08:05:58.659570015Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapError | 0xc00283c700>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-5676c2nc84jf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/81d6e351-7e5e-49b4-85a5-302b276cc8f6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/81d6e351-7e5e-49b4-85a5-302b276cc8f6",
"name": "81d6e351-7e5e-49b4-85a5-302b276cc8f6",
"status": "Failed",
"startTime": "2026-07-23T07:46:53.376501029Z",
"endTime": "2026-07-23T08:05:58.659570015Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-hl8829" for cluster "cluster-candidate-4-21-hl8829"
Unexpected error:
<*fmt.wrapError | 0xc0029b0ac0>:
failed to create NodePool np-hl8829: failed waiting for nodepool="np-hl8829" for cluster "cluster-candidate-4-21-hl8829" in resourcegroup="rg-candidate-4-21-tv28gtjd9f9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d4916d8a-cfac-4b24-ba00-5346b085835c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d4916d8a-cfac-4b24-ba00-5346b085835c",
"name": "d4916d8a-cfac-4b24-ba00-5346b085835c",
"status": "Failed",
"startTime": "2026-07-23T04:16:36.200454364Z",
"endTime": "2026-07-23T04:35:50.806314695Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-hl8829" for cluster "cluster-candidate-4-21-hl8829"
Unexpected error:
<*fmt.wrapError | 0xc0029b0ac0>:
failed to create NodePool np-hl8829: failed waiting for nodepool="np-hl8829" for cluster "cluster-candidate-4-21-hl8829" in resourcegroup="rg-candidate-4-21-tv28gtjd9f9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d4916d8a-cfac-4b24-ba00-5346b085835c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d4916d8a-cfac-4b24-ba00-5346b085835c",
"name": "d4916d8a-cfac-4b24-ba00-5346b085835c",
"status": "Failed",
"startTime": "2026-07-23T04:16:36.200454364Z",
"endTime": "2026-07-23T04:35:50.806314695Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (5)
Affected runs (7)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 4 | 6.67%4 of 60 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsJul 19: 0 · Jul 20: 82 · Jul 21: 42 · Jul 22: 1 · Jul 23: 0 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0xc0006e58a0>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-rp6jsc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/174515fd-ae84-46d9-b491-3d03c175e06c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/174515fd-ae84-46d9-b491-3d03c175e06c",
"name": "174515fd-ae84-46d9-b491-3d03c175e06c",
"status": "Failed",
"startTime": "2026-07-22T12:40:43.885322535Z",
"endTime": "2026-07-22T12:59:47.104093689Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:112]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
<*fmt.wrapError | 0xc0006e58a0>:
failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-rp6jsc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/174515fd-ae84-46d9-b491-3d03c175e06c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/174515fd-ae84-46d9-b491-3d03c175e06c",
"name": "174515fd-ae84-46d9-b491-3d03c175e06c",
"status": "Failed",
"startTime": "2026-07-22T12:40:43.885322535Z",
"endTime": "2026-07-22T12:59:47.104093689Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:159]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0xc000f9e360>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-htsfbf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c99612e5-ef65-4612-87de-9bd468c43288
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c99612e5-ef65-4612-87de-9bd468c43288",
"name": "c99612e5-ef65-4612-87de-9bd468c43288",
"status": "Failed",
"startTime": "2026-07-21T06:15:23.87500457Z",
"endTime": "2026-07-21T06:34:30.805495615Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:159]: failed to create HCP cluster for patch-tags test
Unexpected error:
<*fmt.wrapError | 0xc000f9e360>:
failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-htsfbf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c99612e5-ef65-4612-87de-9bd468c43288
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c99612e5-ef65-4612-87de-9bd468c43288",
"name": "c99612e5-ef65-4612-87de-9bd468c43288",
"status": "Failed",
"startTime": "2026-07-21T06:15:23.87500457Z",
"endTime": "2026-07-21T06:34:30.805495615Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:83]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0xc000e7c300>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-n5wk42" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b7f26b4b-723b-4092-913d-37ce64f501e1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b7f26b4b-723b-4092-913d-37ce64f501e1",
"name": "b7f26b4b-723b-4092-913d-37ce64f501e1",
"status": "Failed",
"startTime": "2026-07-21T05:32:09.443545349Z",
"endTime": "2026-07-21T05:51:18.672656967Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:83]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapError | 0xc000e7c300>:
failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-n5wk42" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b7f26b4b-723b-4092-913d-37ce64f501e1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b7f26b4b-723b-4092-913d-37ce64f501e1",
"name": "b7f26b4b-723b-4092-913d-37ce64f501e1",
"status": "Failed",
"startTime": "2026-07-21T05:32:09.443545349Z",
"endTime": "2026-07-21T05:51:18.672656967Z",
"error": {
"code": "InternalServerError",
"message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (42)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShiftfull failure pattern: ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 4 | 6.67%4 of 60 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsJul 19: 0 · Jul 20: 0 · Jul 21: 8 · Jul 22: 2 · Jul 23: 1 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0xc0001e0aa0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-9n27h62csfxn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/032e5270-928f-475d-9dae-95082a8a0af4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/032e5270-928f-475d-9dae-95082a8a0af4",
"name": "032e5270-928f-475d-9dae-95082a8a0af4",
"status": "Failed",
"startTime": "2026-07-23T23:04:31.918481763Z",
"endTime": "2026-07-23T23:23:43.694668207Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0xc0001e0aa0>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-9n27h62csfxn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/032e5270-928f-475d-9dae-95082a8a0af4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/032e5270-928f-475d-9dae-95082a8a0af4",
"name": "032e5270-928f-475d-9dae-95082a8a0af4",
"status": "Failed",
"startTime": "2026-07-23T23:04:31.918481763Z",
"endTime": "2026-07-23T23:23:43.694668207Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0xc0005871c0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-sj5lp4s8hdhx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/913f1b53-79a1-4c56-90f3-f94f835b8c75
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/913f1b53-79a1-4c56-90f3-f94f835b8c75",
"name": "913f1b53-79a1-4c56-90f3-f94f835b8c75",
"status": "Failed",
"startTime": "2026-07-22T02:04:09.655224875Z",
"endTime": "2026-07-22T02:23:21.611762653Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
<*fmt.wrapError | 0xc0005871c0>:
failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-sj5lp4s8hdhx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/913f1b53-79a1-4c56-90f3-f94f835b8c75
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/913f1b53-79a1-4c56-90f3-f94f835b8c75",
"name": "913f1b53-79a1-4c56-90f3-f94f835b8c75",
"status": "Failed",
"startTime": "2026-07-22T02:04:09.655224875Z",
"endTime": "2026-07-22T02:23:21.611762653Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-20-qnfrfst59rxk/cluster-candidate-4-20-z9f7qm should provision
Unexpected error:
<*fmt.wrapError | 0xc000b10b60>:
failed to create HCP cluster cluster-candidate-4-20-z9f7qm: failed waiting for cluster="cluster-candidate-4-20-z9f7qm" in resourcegroup="rg-candidate-4-20-qnfrfst59rxk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/952533d9-7ad3-4b91-b8b8-23d93e01328b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/952533d9-7ad3-4b91-b8b8-23d93e01328b",
"name": "952533d9-7ad3-4b91-b8b8-23d93e01328b",
"status": "Failed",
"startTime": "2026-07-22T02:05:40.15224731Z",
"endTime": "2026-07-22T02:24:41.285079074Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-20-qnfrfst59rxk/cluster-candidate-4-20-z9f7qm should provision
Unexpected error:
<*fmt.wrapError | 0xc000b10b60>:
failed to create HCP cluster cluster-candidate-4-20-z9f7qm: failed waiting for cluster="cluster-candidate-4-20-z9f7qm" in resourcegroup="rg-candidate-4-20-qnfrfst59rxk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/952533d9-7ad3-4b91-b8b8-23d93e01328b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/952533d9-7ad3-4b91-b8b8-23d93e01328b",
"name": "952533d9-7ad3-4b91-b8b8-23d93e01328b",
"status": "Failed",
"startTime": "2026-07-22T02:05:40.15224731Z",
"endTime": "2026-07-22T02:24:41.285079074Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (11)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Cluster provisioning failedfull failure pattern: Cluster provisioning failed Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 4 | 6.67%4 of 60 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsJul 19: 0 · Jul 20: 3 · Jul 21: 1 · Jul 22: 0 · Jul 23: 0 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Cluster provisioning failed Contributing tests (1)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 4 | 6.67%4 of 60 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsJul 19: 0 · Jul 20: 3 · Jul 21: 1 · Jul 22: 0 · Jul 23: 0 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
<*fmt.wrapErrors | 0xc00098e570>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-t84nc4" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-t84nc4, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
<*fmt.wrapErrors | 0xc00098e570>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-t84nc4" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-t84nc4, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
<*fmt.wrapErrors | 0xc000fee390>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-d2dn29" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-d2dn29, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
<*fmt.wrapErrors | 0xc000fee390>:
failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-d2dn29" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-d2dn29, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-5-0-cpmsgk/cluster-nightly-5-0-vxt256 should provision
Unexpected error:
<*fmt.wrapErrors | 0xc00189f830>:
failed to create HCP cluster cluster-nightly-5-0-vxt256, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-vxt256 in resource group rg-nightly-5-0-cpmsgk, error: failed waiting for cluster="cluster-nightly-5-0-vxt256" in resourcegroup="rg-nightly-5-0-cpmsgk" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-vxt256 in resource group rg-nightly-5-0-cpmsgk, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-5-0-cpmsgk/cluster-nightly-5-0-vxt256 should provision
Unexpected error:
<*fmt.wrapErrors | 0xc00189f830>:
failed to create HCP cluster cluster-nightly-5-0-vxt256, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-vxt256 in resource group rg-nightly-5-0-cpmsgk, error: failed waiting for cluster="cluster-nightly-5-0-vxt256" in resourcegroup="rg-nightly-5-0-cpmsgk" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-vxt256 in resource group rg-nightly-5-0-cpmsgk, error: context deadline exceeded
...
occurredContributing tests (2)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceededfull failure pattern: DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 3.33%2 of 60 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsJul 19: 0 · Jul 20: 1 · Jul 21: 0 · Jul 22: 0 · Jul 23: 0 · Jul 24: 0 · Jul 25: 5 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
<*errors.joinError | 0xc000ab02a0>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="private-keyvault-hzd8f756pkjx", caused by: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-hzd8f756pkjx" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster private-kv-cluster in resource group private-keyvault-hzd8f756pkjx, error: context deadline exceeded, error: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-hzd8f756pkjx" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster private-kv-cluster in resource group private-keyvault-hzd8f756pkjx, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
<*errors.joinError | 0xc000ab02a0>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="private-keyvault-hzd8f756pkjx", caused by: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-hzd8f756pkjx" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster private-kv-cluster in resource group private-keyvault-hzd8f756pkjx, error: context deadline exceeded, error: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-hzd8f756pkjx" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster private-kv-cluster in resource group private-keyvault-hzd8f756pkjx, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
<*errors.joinError | 0xc00172e378>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv", caused by: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-hszcm2jz5pbv, error: context deadline exceeded, error: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-hszcm2jz5pbv, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
<*errors.joinError | 0xc00172e378>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv", caused by: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-hszcm2jz5pbv, error: context deadline exceeded, error: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-hszcm2jz5pbv, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
<*errors.joinError | 0xc000fc0090>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="clusternp128-nzwfdf99d2bd", caused by: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-nzwfdf99d2bd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-nzwfdf99d2bd, error: context deadline exceeded, error: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-nzwfdf99d2bd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-nzwfdf99d2bd, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
<*errors.joinError | 0xc000fc0090>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="clusternp128-nzwfdf99d2bd", caused by: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-nzwfdf99d2bd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-nzwfdf99d2bd, error: context deadline exceeded, error: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-nzwfdf99d2bd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-nzwfdf99d2bd, error: context deadline exceeded
...
occurredContributing tests (6)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | e2e | 2 | 3.33%2 of 60 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 2 day(s) | Show trend detailsJul 19: 0 · Jul 20: 1 · Jul 21: 0 · Jul 22: 1 · Jul 23: 0 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:186]: failed to verify simple web app runs on cluster "cluster-nightly-4-23-dpbrz9"
Unexpected error:
<*fmt.wrapError | 0xc000fbe3e0>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-zszzt.apps.aro.x0o8h6x6l7k9u1f.e0w5.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:186]: failed to verify simple web app runs on cluster "cluster-nightly-4-23-dpbrz9"
Unexpected error:
<*fmt.wrapError | 0xc000fbe3e0>:
[strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-zszzt.apps.aro.x0o8h6x6l7k9u1f.e0w5.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0xc001a0b110>:
VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-sample-app-7gqvj.apps.aro.cilium-cl.gb92.westeurope.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:170]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0xc001a0b110>:
VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-sample-app-7gqvj.apps.aro.cilium-cl.gb92.westeurope.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DeleteResourceGroup timed out after <minutes> minutes; context deadline exceededfull failure pattern: DeleteResourceGroup timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 1.67%1 of 60 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsJul 19: 0 · Jul 20: 0 · Jul 21: 0 · Jul 22: 0 · Jul 23: 0 · Jul 24: 0 · Jul 25: 1 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:145]: failed to delete customer resource group "cx-rg-cluster-jlqwkzs7jqhj"
Unexpected error:
<*fmt.wrapErrors | 0xc001296270>:
failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-jlqwkzs7jqhj, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:145]: failed to delete customer resource group "cx-rg-cluster-jlqwkzs7jqhj"
Unexpected error:
<*fmt.wrapErrors | 0xc001296270>:
failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-jlqwkzs7jqhj, error: context deadline exceeded
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail code DenyAssignmentAuthorizationFailed; detail message Access denied by deny assignment for action "Microsoft.Compute/virtualMachines/retri...full failure pattern: ERROR CODE: InternalServerError; detail code DenyAssignmentAuthorizationFailed; detail message Access denied by deny assignment for action "Microsoft.Compute/virtualMachines/retrieveBootDiagnosticsData/action".; provider Microsoft.Authorization Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.67%1 of 60 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 19: 0 · Jul 20: 0 · Jul 21: 0 · Jul 22: 1 · Jul 23: 0 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc00003c078>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-xxxldl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9",
"name": "9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9",
"status": "Failed",
"startTime": "2026-07-22T08:31:33.476055825Z",
"endTime": "2026-07-22T08:50:37.393817181Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
encountered errors downloading console logs:
- VM "cilium-cluster-cilium-np-8skrs-nxzhn": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-8skrs-nxzhn": POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-nxzhn/retrieveBootDiagnosticsData
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: DenyAssignmentAuthorizationFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "DenyAssignmentAuthorizationFailed",
"message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' has permission to perform action 'Microsoft.Compute/virtualMachines/retrieveBootDiagnosticsData/action' on scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-nxzhn'; however, the access is denied because of the deny assignment with name '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Authorization/denyAssignments/4e1d5354-563b-53f0-a663-9624ce82fcc6' and Id '4e1d5354563b53f0a6639624ce82fcc6' at scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed'."
}
}
--------------------------------------------------------------------------------
- VM "cilium-cluster-cilium-np-8skrs-wwbw4": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-8skrs-wwbw4": POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-wwbw4/retrieveBootDiagnosticsData
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: DenyAssignmentAuthorizationFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "DenyAssignmentAuthorizationFailed",
"message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' has permission to perform action 'Microsoft.Compute/virtualMachines/retrieveBootDiagnosticsData/action' on scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-wwbw4'; however, the access is denied because of the deny assignment with name '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Authorization/denyAssignments/4e1d5354-563b-53f0-a663-9624ce82fcc6' and Id '4e1d5354563b53f0a6639624ce82fcc6' at scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed'."
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc00003c078>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-xxxldl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9",
"name": "9c4f6c2a-fe7f-4a6d-b9a4-e7fb6b5215d9",
"status": "Failed",
"startTime": "2026-07-22T08:31:33.476055825Z",
"endTime": "2026-07-22T08:50:37.393817181Z",
"error": {
"code": "InternalServerError",
"message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
}
}
--------------------------------------------------------------------------------
encountered errors downloading console logs:
- VM "cilium-cluster-cilium-np-8skrs-nxzhn": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-8skrs-nxzhn": POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-nxzhn/retrieveBootDiagnosticsData
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: DenyAssignmentAuthorizationFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "DenyAssignmentAuthorizationFailed",
"message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' has permission to perform action 'Microsoft.Compute/virtualMachines/retrieveBootDiagnosticsData/action' on scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-nxzhn'; however, the access is denied because of the deny assignment with name '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Authorization/denyAssignments/4e1d5354-563b-53f0-a663-9624ce82fcc6' and Id '4e1d5354563b53f0a6639624ce82fcc6' at scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed'."
}
}
--------------------------------------------------------------------------------
- VM "cilium-cluster-cilium-np-8skrs-wwbw4": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-8skrs-wwbw4": POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-wwbw4/retrieveBootDiagnosticsData
--------------------------------------------------------------------------------
RESPONSE 403: 403 Forbidden
ERROR CODE: DenyAssignmentAuthorizationFailed
--------------------------------------------------------------------------------
{
"error": {
"code": "DenyAssignmentAuthorizationFailed",
"message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' has permission to perform action 'Microsoft.Compute/virtualMachines/retrieveBootDiagnosticsData/action' on scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8skrs-wwbw4'; however, the access is denied because of the deny assignment with name '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed/providers/Microsoft.Authorization/denyAssignments/4e1d5354-563b-53f0-a663-9624ce82fcc6' and Id '4e1d5354563b53f0a6639624ce82fcc6' at scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-xxxldl--managed'."
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider...full failure pattern: ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.67%1 of 60 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 19: 0 · Jul 20: 0 · Jul 21: 0 · Jul 22: 0 · Jul 23: 0 · Jul 24: 0 · Jul 25: 1 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-nightly-4-23-bvp8bvjdldql/cluster-nightly-4-23-6252tm should provision
Unexpected error:
<*fmt.wrapError | 0xc000b819a0>:
failed to create HCP cluster cluster-nightly-4-23-6252tm: failed waiting for cluster="cluster-nightly-4-23-6252tm" in resourcegroup="rg-nightly-4-23-bvp8bvjdldql" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a07651a-a9b8-414c-8974-3522743736b5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a07651a-a9b8-414c-8974-3522743736b5",
"name": "7a07651a-a9b8-414c-8974-3522743736b5",
"status": "Failed",
"startTime": "2026-07-25T23:30:22.26941828Z",
"endTime": "2026-07-25T23:49:39.833761299Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-nightly-4-23-bvp8bvjdldql/cluster-nightly-4-23-6252tm should provision
Unexpected error:
<*fmt.wrapError | 0xc000b819a0>:
failed to create HCP cluster cluster-nightly-4-23-6252tm: failed waiting for cluster="cluster-nightly-4-23-6252tm" in resourcegroup="rg-nightly-4-23-bvp8bvjdldql" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a07651a-a9b8-414c-8974-3522743736b5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a07651a-a9b8-414c-8974-3522743736b5",
"name": "7a07651a-a9b8-414c-8974-3522743736b5",
"status": "Failed",
"startTime": "2026-07-25T23:30:22.26941828Z",
"endTime": "2026-07-25T23:49:39.833761299Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; host...full failure pattern: ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.67%1 of 60 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 19: 0 · Jul 20: 0 · Jul 21: 1 · Jul 22: 0 · Jul 23: 0 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-8r5shn/cluster-candidate-5-0-wj576z should provision
Unexpected error:
<*fmt.wrapError | 0xc000cdc260>:
failed to create HCP cluster cluster-candidate-5-0-wj576z: failed waiting for cluster="cluster-candidate-5-0-wj576z" in resourcegroup="rg-candidate-5-0-8r5shn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3119b6a0-b91c-4658-8b22-178014841c9c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3119b6a0-b91c-4658-8b22-178014841c9c",
"name": "3119b6a0-b91c-4658-8b22-178014841c9c",
"status": "Failed",
"startTime": "2026-07-21T17:26:18.759928525Z",
"endTime": "2026-07-21T17:45:20.361591697Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-8r5shn/cluster-candidate-5-0-wj576z should provision
Unexpected error:
<*fmt.wrapError | 0xc000cdc260>:
failed to create HCP cluster cluster-candidate-5-0-wj576z: failed waiting for cluster="cluster-candidate-5-0-wj576z" in resourcegroup="rg-candidate-5-0-8r5shn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3119b6a0-b91c-4658-8b22-178014841c9c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/3119b6a0-b91c-4658-8b22-178014841c9c",
"name": "3119b6a0-b91c-4658-8b22-178014841c9c",
"status": "Failed",
"startTime": "2026-07-21T17:26:18.759928525Z",
"endTime": "2026-07-21T17:45:20.361591697Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; host...full failure pattern: ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.67%1 of 60 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 19: 0 · Jul 20: 0 · Jul 21: 1 · Jul 22: 0 · Jul 23: 0 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-22-n6rg6v/cluster-candidate-4-22-zw26wz should provision
Unexpected error:
<*fmt.wrapError | 0xc0009b0c00>:
failed to create HCP cluster cluster-candidate-4-22-zw26wz: failed waiting for cluster="cluster-candidate-4-22-zw26wz" in resourcegroup="rg-candidate-4-22-n6rg6v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6d83515e-bf6d-403d-8393-495c72f0b7bb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6d83515e-bf6d-403d-8393-495c72f0b7bb",
"name": "6d83515e-bf6d-403d-8393-495c72f0b7bb",
"status": "Failed",
"startTime": "2026-07-21T17:26:00.417709536Z",
"endTime": "2026-07-21T17:45:01.077714445Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-22-n6rg6v/cluster-candidate-4-22-zw26wz should provision
Unexpected error:
<*fmt.wrapError | 0xc0009b0c00>:
failed to create HCP cluster cluster-candidate-4-22-zw26wz: failed waiting for cluster="cluster-candidate-4-22-zw26wz" in resourcegroup="rg-candidate-4-22-n6rg6v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6d83515e-bf6d-403d-8393-495c72f0b7bb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/6d83515e-bf6d-403d-8393-495c72f0b7bb",
"name": "6d83515e-bf6d-403d-8393-495c72f0b7bb",
"status": "Failed",
"startTime": "2026-07-21T17:26:00.417709536Z",
"endTime": "2026-07-21T17:45:01.077714445Z",
"error": {
"code": "InternalServerError",
"message": "[hypershiftHostedCluster] hosted cluster is not available: KASLoadBalancerNotReachable: APIServer external route not admitted; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: InvalidTemplateDeployment; provider Microsoft.Networkfull failure pattern: ERROR CODE: InvalidTemplateDeployment; provider Microsoft.Network Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.67%1 of 60 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 19: 0 · Jul 20: 1 · Jul 21: 0 · Jul 22: 0 · Jul 23: 0 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:131]: failed to deploy test VM
Unexpected error:
<*fmt.wrapError | 0xc00155ef00>:
failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-l96fjf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-l96fjf/providers/Microsoft.Resources/deployments/test-vm
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidTemplateDeployment
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidTemplateDeployment",
"message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Network/publicIPAddresses (2022-07-01)' reported preflight validation errors. Tracking id is '3a84a790-e152-4d73-a198-bf0f9d6b6d4a'. See inner errors for details.",
"details": [
{
"code": "ResourceCountExceedsLimitDueToTemplate",
"message": "Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 300 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 300 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress",
"details": []
}
]
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:131]: failed to deploy test VM
Unexpected error:
<*fmt.wrapError | 0xc00155ef00>:
failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-l96fjf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-l96fjf/providers/Microsoft.Resources/deployments/test-vm
--------------------------------------------------------------------------------
RESPONSE 400: 400 Bad Request
ERROR CODE: InvalidTemplateDeployment
--------------------------------------------------------------------------------
{
"error": {
"code": "InvalidTemplateDeployment",
"message": "The template deployment 'test-vm' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.Network/publicIPAddresses (2022-07-01)' reported preflight validation errors. Tracking id is '3a84a790-e152-4d73-a198-bf0f9d6b6d4a'. See inner errors for details.",
"details": [
{
"code": "ResourceCountExceedsLimitDueToTemplate",
"message": "Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 300 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 300 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress",
"details": []
}
]
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
add password after 2 attempts; last attempt error: error status code received from the API; polling error: error status code received from the APIfull failure pattern: add password after 2 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 1.67%1 of 60 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 19: 0 · Jul 20: 0 · Jul 21: 0 · Jul 22: 0 · Jul 23: 1 · Jul 24: 0 · Jul 25: 0 · Jul 26: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:132]: failed to add password to app registration
Unexpected error:
<*fmt.wrapErrors | 0xc0013a57d0>:
add password after 2 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:132]: failed to add password to app registration
Unexpected error:
<*fmt.wrapErrors | 0xc0013a57d0>:
add password after 2 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||