CIHealth
<
Single Day: Sep 30▾
Last 7 Days Weekly: Sep 28 - Oct 4 Sprint: Sep 28 - Oct 11 Single Day: Sep 30
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 51 · Failures matched: 145 · Total runs: 98 · Runs affected: 48

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRClusterVersion had no completed update entry in history
full failure pattern:
ClusterVersion had no completed update entry in history
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:487]: ClusterVersion had no completed update entry in history
Expected
    <string>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:487]: ClusterVersion had no completed update entry in history
Expected
    <string>: ...
not to be empty
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71812105167861478592512
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRDeserialization Error: no output from command
full failure pattern:
Deserialization Error: no output from command
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:409 +0x87 fp=0xc0000d9e90 sp=0xc0000d9e20 pc=0xbff15a7
github.com/Azure/ARO-HCP/test/util/framework.runResourceGroupCleanup.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:410 +0x7e fp=0xc0000d9f78 sp=0xc0000d9e90 pc=0xbff12be
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50 fp=0xc0000d9fe0 sp=0xc0000d9f78 pc=0x6f51610
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000d9fe8 sp=0xc0000d9fe0 pc=0x5439fc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 3248
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95

goroutine 3283 gp=0xc0011bfdc0 m=nil [select, 2 minutes]:
runtime.gopark(0xc000e65f48?, 0x2?, 0xc0?, 0x5d?, 0xc000e65ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000df4d78 sp=0xc000df4d58 pc=0x54317ae
runtime.selectgo(0xc000df4f48, 0xc000e65ef0, 0xc00191c0c0?, 0x0, 0xc0028017d0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000df4eb8 sp=0xc000df4d78 pc=0x540c3b7
net/http.(*persistConn).writeLoop(0xc0013e3b00)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc000df4fc8 sp=0xc000df4eb8 pc=0x5803227
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc000df4fe0 sp=0xc000df4fc8 pc=0x57ffec5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000df4fe8 sp=0xc000df4fe0 pc=0x5439fc1
created by net/http.(*Transport).dialConn in goroutine 3249
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 3232 gp=0xc0002616c0 m=nil [select]:
runtime.gopark(0xc001a7f768?, 0x2?, 0x40?, 0x0?, 0xc001a7f75c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001a7f5e8 sp=0xc001a7f5c8 pc=0x54317ae
runtime.selectgo(0xc001a7f768, 0xc001a7f758, 0xc001a7f788?, 0x0, 0xf83b8ce?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc001a7f728 sp=0xc001a7f5e8 pc=0x540c3b7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x108bfe40, 0xc00066ad20}, 0xf7fc714?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.23.0/internal/shared/shared.go:36 +0x76 fp=0xc001a7f798 sp=0xc001a7f728 pc=0x5c1c996
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x10919ee0, {0x108bfe40, 0xc00066ad20}, 0x542ed93)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.23.0/runtime/poller.go:284 +0x732 fp=0xc001a7fa98 sp=0xc001a7f798 pc=0x5e69a92
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x108bfe40, 0xc00066ad20}, 0xc00104bae8, {0xc000f74aa0, 0x1c}, {0xc0018c1ef0, 0x15})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:591 +0x365 fp=0xc001a7fc20 sp=0xc001a7fa98 pc=0xbfaf265
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x108bfe40?, 0xc00066ad20?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:628 +0x72 fp=0xc001a7fcc8 sp=0xc001a7fc20 pc=0xbfafa12
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x108bfe40?, 0xc00066ad20?}, 0xc00066ad30?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc001a7fd20 sp=0xc001a7fcc8 pc=0x5bd1a22
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x108bfe40, 0xc00066ad20}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc001a7fe68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc001a7fd98 sp=0xc001a7fd20 pc=0x5bcface
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x108bfdd0, 0xc000f19220}, 0xc00104bae8, {0xc000f74aa0, 0x1c}, {0xc0018c1ef0, 0x15}, 0x15d3ef79800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:626 +0x285 fp=0xc001a7fef0 sp=0xc001a7fd98 pc=0xbfaf785
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:799 +0x7e fp=0xc001a7ff78 sp=0xc001a7fef0 pc=0xbfb219e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50 fp=0xc001a7ffe0 sp=0xc001a7ff78 pc=0x6f51610
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001a7ffe8 sp=0xc001a7ffe0 pc=0x5439fc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 3273
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x543bde3
rdx    0x0
rdi    0x169679d0
rsi    0x80
rbp    0x7ffc297c0df0
rsp    0x7ffc297c0da8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x3
r13    0x1
r14    0x16965440
r15    0x5
rip    0x543bde1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:409 +0x87 fp=0xc0000d9e90 sp=0xc0000d9e20 pc=0xbff15a7
github.com/Azure/ARO-HCP/test/util/framework.runResourceGroupCleanup.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:410 +0x7e fp=0xc0000d9f78 sp=0xc0000d9e90 pc=0xbff12be
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50 fp=0xc0000d9fe0 sp=0xc0000d9f78 pc=0x6f51610
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc0000d9fe8 sp=0xc0000d9fe0 pc=0x5439fc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 3248
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95

goroutine 3283 gp=0xc0011bfdc0 m=nil [select, 2 minutes]:
runtime.gopark(0xc000e65f48?, 0x2?, 0xc0?, 0x5d?, 0xc000e65ef4?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc000df4d78 sp=0xc000df4d58 pc=0x54317ae
runtime.selectgo(0xc000df4f48, 0xc000e65ef0, 0xc00191c0c0?, 0x0, 0xc0028017d0?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc000df4eb8 sp=0xc000df4d78 pc=0x540c3b7
net/http.(*persistConn).writeLoop(0xc0013e3b00)
	/usr/lib/golang/src/net/http/transport.go:2600 +0xe7 fp=0xc000df4fc8 sp=0xc000df4eb8 pc=0x5803227
net/http.(*Transport).dialConn.gowrap3()
	/usr/lib/golang/src/net/http/transport.go:1948 +0x25 fp=0xc000df4fe0 sp=0xc000df4fc8 pc=0x57ffec5
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc000df4fe8 sp=0xc000df4fe0 pc=0x5439fc1
created by net/http.(*Transport).dialConn in goroutine 3249
	/usr/lib/golang/src/net/http/transport.go:1948 +0x17a5

goroutine 3232 gp=0xc0002616c0 m=nil [select]:
runtime.gopark(0xc001a7f768?, 0x2?, 0x40?, 0x0?, 0xc001a7f75c?)
	/usr/lib/golang/src/runtime/proc.go:460 +0xce fp=0xc001a7f5e8 sp=0xc001a7f5c8 pc=0x54317ae
runtime.selectgo(0xc001a7f768, 0xc001a7f758, 0xc001a7f788?, 0x0, 0xf83b8ce?, 0x1)
	/usr/lib/golang/src/runtime/select.go:351 +0x8b7 fp=0xc001a7f728 sp=0xc001a7f5e8 pc=0x540c3b7
github.com/Azure/azure-sdk-for-go/sdk/azcore/internal/shared.Delay({0x108bfe40, 0xc00066ad20}, 0xf7fc714?)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.23.0/internal/shared/shared.go:36 +0x76 fp=0xc001a7f798 sp=0xc001a7f728 pc=0x5c1c996
github.com/Azure/azure-sdk-for-go/sdk/azcore/runtime.(*Poller[...]).PollUntilDone(0x10919ee0, {0x108bfe40, 0xc00066ad20}, 0x542ed93)
	/go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/azcore@v1.23.0/runtime/poller.go:284 +0x732 fp=0xc001a7fa98 sp=0xc001a7f798 pc=0x5e69a92
github.com/Azure/ARO-HCP/test/util/framework.deleteHCPClusterAttempt({0x108bfe40, 0xc00066ad20}, 0xc00104bae8, {0xc000f74aa0, 0x1c}, {0xc0018c1ef0, 0x15})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:591 +0x365 fp=0xc001a7fc20 sp=0xc001a7fa98 pc=0xbfaf265
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610.func1({0x108bfe40?, 0xc00066ad20?})
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:628 +0x72 fp=0xc001a7fcc8 sp=0xc001a7fc20 pc=0xbfafa12
k8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext({0x108bfe40?, 0xc00066ad20?}, 0xc00066ad30?)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/wait.go:159 +0x62 fp=0xc001a7fd20 sp=0xc001a7fcc8 pc=0x5bd1a22
k8s.io/apimachinery/pkg/util/wait.ExponentialBackoffWithContext({0x108bfe40, 0xc00066ad20}, {0xdf8475800, 0x4000000000000000, 0x3fb999999999999a, 0x5, 0x0}, 0xc001a7fe68)
	/go/pkg/mod/k8s.io/apimachinery@v0.35.3/pkg/util/wait/backoff.go:501 +0xae fp=0xc001a7fd98 sp=0xc001a7fd20 pc=0x5bcface
github.com/Azure/ARO-HCP/test/util/framework.DeleteHCPCluster20240610({0x108bfdd0, 0xc000f19220}, 0xc00104bae8, {0xc000f74aa0, 0x1c}, {0xc0018c1ef0, 0x15}, 0x15d3ef79800)
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:626 +0x285 fp=0xc001a7fef0 sp=0xc001a7fd98 pc=0xbfaf785
github.com/Azure/ARO-HCP/test/util/framework.DeleteAllHCPClusters20240610.func1()
	/opt/app-root/src/github.com/Azure/ARO-HCP/test/util/framework/helpers_v20240610preview.go:799 +0x7e fp=0xc001a7ff78 sp=0xc001a7fef0 pc=0xbfb219e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:93 +0x50 fp=0xc001a7ffe0 sp=0xc001a7ff78 pc=0x6f51610
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1693 +0x1 fp=0xc001a7ffe8 sp=0xc001a7ffe0 pc=0x5439fc1
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 3273
	/go/pkg/mod/golang.org/x/sync@v0.22.0/errgroup/errgroup.go:78 +0x95

rax    0xca
rbx    0x0
rcx    0x543bde3
rdx    0x0
rdi    0x169679d0
rsi    0x80
rbp    0x7ffc297c0df0
rsp    0x7ffc297c0da8
r8     0x0
r9     0x0
r10    0x0
r11    0x286
r12    0x3
r13    0x1
r14    0x16965440
r15    0x5
rip    0x543bde1
rflags 0x286
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72612105187789510283264
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InvalidRequestContent; detail message Invalid value: "/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIde...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi": containerRegistry requires cluster version 4.22.0 or above; provider Microsoft.ManagedIdentity
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:242]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc000b4a880>: 
    failed starting cluster creation "acr-pull-create" in resourcegroup="acr-pull-create-v5r8zv9r2jvq": PUT https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-v5r8zv9r2jvq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/acr-pull-create
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-v5r8zv9r2jvq/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi\": containerRegistry requires cluster version 4.22.0 or above",
        "target": "properties.platform.containerRegistry.managedIdentity"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:242]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0xc000b4a880>: 
    failed starting cluster creation "acr-pull-create" in resourcegroup="acr-pull-create-v5r8zv9r2jvq": PUT https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-v5r8zv9r2jvq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/acr-pull-create
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/acr-pull-create-v5r8zv9r2jvq/providers/Microsoft.ManagedIdentity/userAssignedIdentities/acr-pull-mi\": containerRegistry requires cluster version 4.22.0 or above",
        "target": "properties.platform.containerRegistry.managedIdentity"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72692105294127649263616
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InvalidRequestContent; detail message Invalid value: "<hex>": KMS key version rotation requires cluster version 4.22.0 or above; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<hex>": KMS key version rotation requires cluster version 4.22.0 or above; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:191]: failed to update cluster with new KMS key
Unexpected error:
    <*exported.ResponseError | 0xc000ece360>: 
    PATCH https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kms-key-rotate-s7rqrhtvbcff/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/kms-key-rotate-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"484184ba23584b37813d670c444e552b\": KMS key version rotation requires cluster version 4.22.0 or above",
        "target": "properties.etcd.dataEncryption.customerManaged.kms.activeKey.version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:191]: failed to update cluster with new KMS key
Unexpected error:
    <*exported.ResponseError | 0xc000ece360>: 
    PATCH https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kms-key-rotate-s7rqrhtvbcff/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/kms-key-rotate-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"484184ba23584b37813d670c444e552b\": KMS key version rotation requires cluster version 4.22.0 or above",
        "target": "properties.etcd.dataEncryption.customerManaged.kms.activeKey.version"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72692105294127649263616
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: InvalidRequestContent; detail message Invalid value: "Private": not supported when customerProperties.version.id is below 4.22; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "Private": not supported when customerProperties.version.id is below 4.22; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:96]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000c48700>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-7mqkjzxbfvhm": PUT https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-7mqkjzxbfvhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"Private\": not supported when customerProperties.version.id is below 4.22",
        "target": "properties.api.visibility"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:96]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000c48700>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-7mqkjzxbfvhm": PUT https://rp.j9263616.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-7mqkjzxbfvhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"Private\": not supported when customerProperties.version.id is below 4.22",
        "target": "properties.api.visibility"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72692105294127649263616
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyVMOSDiskCustomerEncryption(nodePool=<nodepool>) failed: no VMs found for nodepool <nodepool> in managed resource group <resource-group>
full failure pattern:
VerifyVMOSDiskCustomerEncryption(nodePool=<nodepool>) failed: no VMs found for nodepool <nodepool> in managed resource group <resource-group>
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:175]: cluster verification failed
Unexpected error:
    <*errors.joinError | 0xc0015b4cc0>: 
    VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: no VMs found for nodepool des-np in managed resource group des-encrypt-gngwklk9wp6n--managed
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:175]: cluster verification failed
Unexpected error:
    <*errors.joinError | 0xc0015b4cc0>: 
    VerifyVMOSDiskCustomerEncryption(nodePool=des-np) failed: no VMs found for nodepool des-np in managed resource group des-encrypt-gngwklk9wp6n--managed
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71432105382781734359040
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] KubeDaemonSetMisScheduled fired
full failure pattern:
alert [svc] KubeDaemonSetMisScheduled fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:16:20Z
  Ended:       2026-09-30T17:27:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="velero", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:16:20Z
  Ended:       2026-09-30T17:27:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T17:16:20Z
  Ended:       2026-09-30T17:26:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet arobit/arobit-forwarder are running where they are not supposed to run.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T17:16:20Z
  Ended:       2026-09-30T17:26:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetMisScheduled", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="node-agent", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="velero", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: 1 Pods of DaemonSet velero/node-agent are running where they are not supposed to run.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDaemonSetMisScheduled does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782105327789455446016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [svc] KubeNodeUnreachable fired
full failure pattern:
alert [svc] KubeNodeUnreachable fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:15:29Z
  Ended:       2026-09-30T17:26:26Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="prometheus", node="aks-userswft1-23433410-vmss000001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft1-23433410-vmss000001 is unreachable and some workloads may be rescheduled.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:16:26Z
  Ended:       2026-09-30T17:26:26Z
  Severity:    Sev3
  Labels:      alertname="KubeNodeUnreachable", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", effect="NoSchedule", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", key="node.kubernetes.io/unreachable", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="prometheus", node="aks-userswft1-23433410-vmss000001", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: aks-userswft1-23433410-vmss000001 is unreachable and some workloads may be rescheduled.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeNodeUnreachable does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782105327789455446016
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to roll out Helm release: resource Deployment/aro-hcp/aro-hcp-backend not ready. status: InProgress, message: Available: 0/2 context deadline exceeded
full failure pattern:
failed to roll out Helm release: resource Deployment/aro-hcp/aro-hcp-backend not ready. status: InProgress, message: Available: 0/2 context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
time=2026-09-30T10:01:25.496Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-09-30T10:01:25.890Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperPoolCert-34 clientId:72671881-beb9-4028-bc7c-47f511281566 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpci01oidcj2121984.z9.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpci01oidcj2121984.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] backupCadenceProfile:testing backupScheduleState:Disabled clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:072ccd06-457a-49cc-a612-ad4dbbc199a3 databaseName:arohcpci01-rp-j2121984 databaseUrl:https://arohcpci01-rp-j2121984.documents.azure.com:443/ kubeApplierCosmosContainerName:Manifests-MC-1 location:canadacentral] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-gq2sycdx/pipeline@sha256:ec35d9956c06798bf0f6c7e4116d8b1c120ff9a5a7cbe4e991d36cae1bea464e replicas:2 resources:map[limits:map[memory:unlimited] requests:map[cpu:700m memory:1Gi]] zoneCount:3] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] logVerbosity:4 maestro:map[sourceEnvironmentIdentifier:arohcpci01] miMock:map[certName:msiMockPoolCert-18 clientId:8c088886-a118-4973-8168-0d7ae1493aa7 clientIdSecretName: principalId:ccbf438f-08ec-4e2b-ae4e-219ce7dc3762 principalIdSecretName: tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-gq2sycdx/pipeline:pull workloadIdentityClientId:7cbf62ec-0a3a-4039-a0c0-7c88aa6f75bf workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:072ccd06-457a-49cc-a612-ad4dbbc199a3 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv]]"
time=2026-09-30T10:01:25.890Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-09-30T10:01:26.074Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-09-30T10:01:26.234Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false
time=2026-09-30T10:01:26.234Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-09-30T10:01:26.298Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-09-30T10:06:31.687Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource Deployment/aro-hcp/aro-hcp-backend not ready. status: InProgress, message: Available: 0/2\ncontext deadline exceeded"
time=2026-09-30T10:06:31.687Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/aro-hcp/aro-hcp-backend not ready. status: InProgress, message: Available: 0/2\ncontext deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.RP.Backend/service/deploy1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69442105224754142121984
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRno running pods found in openshift-kube-apiserver
full failure pattern:
no running pods found in openshift-kube-apiserver
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:226]: Timed out after 300.001s.
KAS→kubelet log path never became reachable from VM inside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:220 with:
no running pods found in openshift-kube-apiserver
Expected
    <string>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:226]: Timed out after 300.001s.
KAS→kubelet log path never became reachable from VM inside the VNet
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_private_nodepool_scaling.go:220 with:
no running pods found in openshift-kube-apiserver
Expected
    <string>: ...
not to be empty
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and successfully scale nodepool replicas1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #65802105349698964426752
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRrun command "e2e-runcommand-1790764865640292876" failed with exit code 28 on VM "<vm>" (output: "000")
full failure pattern:
run command "e2e-runcommand-1790764865640292876" failed with exit code 28 on VM "<vm>" (output: "000")
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.02%1 of 98 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 611.396s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
    <*errors.errorString | 0xc00158e610>: 
    run command "e2e-runcommand-1790764865640292876" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 611.396s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
    <*errors.errorString | 0xc00158e610>: 
    run command "e2e-runcommand-1790764865640292876" failed with exit code 28 on VM "private-ingress-test-vm" (output: "000")
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72202105227431290867712
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert1010.20%10 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 24: 4 · Sep 25: 6 · Sep 26: 1 · Sep 27: 2 · Sep 28: 1 · Sep 29: 7 · Sep 30: 10none
Full failure examples (3)
alert fired 27 time(s) (20 unknown, 7 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-9p7qd", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-9p7qd has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-75htw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-75htw has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-hsnlr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-hsnlr has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-bdq7h", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-bdq7h has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-t7t2f", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-t7t2f has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-c472z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-c472z has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-6j8pl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-6j8pl has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-5vmb4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-5vmb4 has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-z5vtc", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-z5vtc has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-9llk2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-9llk2 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-b79df", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-b79df has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-pgcdv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-pgcdv has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-cm978", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-cm978 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-bb46w", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-bb46w has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-sqbrn", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-sqbrn has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-kmksg", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-kmksg has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-s5k4b", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-s5k4b has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-v55n5", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-v55n5 has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-lxq4s", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-lxq4s has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-30T23:32:33Z
  Ended:       2026-09-30T23:48:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5779712-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a", pod="ovnkube-control-plane-68b744478b-xfjzk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t5kiq11h0qvt8t7eorikel98jokuk8l-p9t6l4h7d3g4h7a/ovnkube-control-plane-68b744478b-xfjzk has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T23:47:21Z
  Ended:       2026-09-30T23:53:25Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2035712-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5ki3fusaq8plth807pisqonnpgbdn6-e4t7z1r7k6m9l4f", pod="kube-state-metrics-hcp-754f855b69-774l4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5ki3fusaq8plth807pisqonnpgbdn6-e4t7z1r7k6m9l4f/kube-state-metrics-hcp-754f855b69-774l4 has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T23:20:44Z
  Ended:       2026-09-30T23:26:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4901760-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5k6opsebogjptm5uab2qhhik4rqbt7-k4w6n5p5s8z3z6k", pod="azure-file-csi-driver-controller-7c869d77d7-z84jf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5k6opsebogjptm5uab2qhhik4rqbt7-k4w6n5p5s8z3z6k/azure-file-csi-driver-controller-7c869d77d7-z84jf has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire10
Affected runs (10)
Date (UTC)Associated PRProw job
PR #65772105424510625779712
PR #71812105423324162035712
PR #69932105417810694901760
PR #65772105370345325203456
PR #69932105342498506280960
PR #71242105331328655298560
PR #72612105310240105828352
PR #71412105297136064139264
PR #69442105295109229645824
PR #69002105244373301071872
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert77.14%7 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 24: 4 · Sep 25: 1 · Sep 26: 2 · Sep 27: 1 · Sep 28: 5 · Sep 29: 6 · Sep 30: 7none
Full failure examples (3)
alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:29:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8920064-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-ptxgs", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-ptxgs has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:29:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8920064-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-rfxj6", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-rfxj6 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:33:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8920064-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-fwc59", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-fwc59 has been in a non-ready state for longer than 5 minutes.
alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:32:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0633472-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-qp5bd", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-qp5bd has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:32:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0633472-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-x78jx", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-x78jx has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:32:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0633472-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-55p2d", region="canadacentral", severity="warning"
  Description: Pod arobit/arobit-forwarder-55p2d has been in a non-ready state for longer than 5 minutes.
alert fired 93 time(s) (25 unknown, 68 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:04:15Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="etcd-2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0e8dbaqbo8t5dt5vdcasi8k08a4q-v2v3z7b8u7q0n1w", pod="kube-state-metrics-hcp-754f855b69-w9pxh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0e8dbaqbo8t5dt5vdcasi8k08a4q-v2v3z7b8u7q0n1w/kube-state-metrics-hcp-754f855b69-w9pxh has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:34:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n", pod="router-79bddd4695-4xzx4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n/router-79bddd4695-4xzx4 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="multus-admission-controller-77fb5c8d8f-hmg8g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/multus-admission-controller-77fb5c8d8f-hmg8g has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:34:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p", pod="router-6c4c4f9b46-9jc7h", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p/router-6c4c4f9b46-9jc7h has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="network-node-identity-7b6f464c54-ttl5f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/network-node-identity-7b6f464c54-ttl5f has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lqugf7p62g2vu15647hciuo7rqe-ephemeral-disk", pod="kube-state-metrics-hcp-754f855b69-5lx7w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lqugf7p62g2vu15647hciuo7rqe-ephemeral-disk/kube-state-metrics-hcp-754f855b69-5lx7w has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f1mqu517if66tcq7vv1883rm6svr2-x0m6x6k6h4c6v2j", pod="kube-state-metrics-hcp-754f855b69-cvfnm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f1mqu517if66tcq7vv1883rm6svr2-x0m6x6k6h4c6v2j/kube-state-metrics-hcp-754f855b69-cvfnm has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f19n96q5slf0m0lsru0ta7ea87dun-h5n4x4c7s1h6n9x", pod="kube-state-metrics-hcp-754f855b69-5ds5m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f19n96q5slf0m0lsru0ta7ea87dun-h5n4x4c7s1h6n9x/kube-state-metrics-hcp-754f855b69-5ds5m has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-30T17:05:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="ovnkube-control-plane-549bdb54d-n62hb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/ovnkube-control-plane-549bdb54d-n62hb has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-30T17:06:13Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0g0m8v1rghh660hh9lh81tcp6q5n-private-ingress", pod="network-node-identity-5469fd6495-rbqgx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0g0m8v1rghh660hh9lh81tcp6q5n-private-ingress/network-node-identity-5469fd6495-rbqgx has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-30T17:06:14Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="kube-controller-manager-59b5946b4c-8dnp4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/kube-controller-manager-59b5946b4c-8dnp4 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-30T17:08:15Z
  Ended:       2026-09-30T17:14:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0nnvckp9ok2ivvajoua5mi6rfcis-e4a2q0w1u4z3v6w", pod="openshift-apiserver-5b6785f9dc-fc7tr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0nnvckp9ok2ivvajoua5mi6rfcis-e4a2q0w1u4z3v6w/openshift-apiserver-5b6785f9dc-fc7tr has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-30T17:08:15Z
  Ended:       2026-09-30T17:14:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f15npkrjd452rtt7ar07bisihfskg-j2c7h4h2c6p4s3p", pod="openshift-apiserver-7dd55877cd-gkcd5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f15npkrjd452rtt7ar07bisihfskg-j2c7h4h2c6p4s3p/openshift-apiserver-7dd55877cd-gkcd5 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-30T17:08:15Z
  Ended:       2026-09-30T17:14:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0uvlpj2ovpuf5b8grf3i9edvpodq-g3v6g1h5e0h5l2j", pod="openshift-apiserver-758c4878d9-7j48d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0uvlpj2ovpuf5b8grf3i9edvpodq-g3v6g1h5e0h5l2j/openshift-apiserver-758c4878d9-7j48d has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-09-30T17:11:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0g0m8v1rghh660hh9lh81tcp6q5n-private-ingress", pod="network-node-identity-5469fd6495-f7zgd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0g0m8v1rghh660hh9lh81tcp6q5n-private-ingress/network-node-identity-5469fd6495-f7zgd has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-09-30T17:11:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="router-8565cb6474-894zb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/router-8565cb6474-894zb has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-09-30T17:11:13Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="kube-apiserver-674f58879-j82fb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/kube-apiserver-674f58879-j82fb has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-09-30T17:11:14Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="network-node-identity-7b6f464c54-gjfm6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/network-node-identity-7b6f464c54-gjfm6 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-09-30T17:11:14Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d", pod="openshift-apiserver-f89c544f9-jcfcs", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0lols5idrpgl73f9vshufog8tnob-e0j9d8p1c1m0o7d/openshift-apiserver-f89c544f9-jcfcs has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-09-30T17:11:14Z
  Ended:       2026-09-30T17:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0suq7elir5mh0aa877skrjk5onf2-t7t6k6q0r6g9t5v", pod="openshift-apiserver-5d4fc6f8c7-68m94", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0suq7elir5mh0aa877skrjk5onf2-t7t6k6q0r6g9t5v/openshift-apiserver-5d4fc6f8c7-68m94 has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-09-30T17:11:14Z
  Ended:       2026-09-30T17:27:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0suq7elir5mh0aa877skrjk5onf2-t7t6k6q0r6g9t5v", pod="kube-apiserver-87c8b5fd9-ph74m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2t5f0suq7elir5mh0aa877skrjk5onf2-t7t6k6q0r6g9t5v/kube-apiserver-87c8b5fd9-ph74m has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-09-30T17:13:16Z
  Ended:       2026-09-30T17:22:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-p75kr", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-p75kr has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:13:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-xfhsn", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-xfhsn has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:13:12Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5446016-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="set-kubelet-parameters-for-scale-vdkwr", region="westus3", severity="warning"
  Description: Pod kube-system/set-kubelet-parameters-for-scale-vdkwr has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire7
Affected runs (7)
Date (UTC)Associated PRProw job
PR #68202105385002328920064
PR #68202105354569830633472
PR #71782105327789455446016
PR #68202105294563651358720
PR #68202105227914915090432
PR #69442105224754142121984
PR #71582105165316576251904
alert [svc] KubeDaemonSetRolloutStuck fired
full failure pattern:
alert [svc] KubeDaemonSetRolloutStuck fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
alert55.10%5 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2 · Sep 29: 0 · Sep 30: 5none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:39:30Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j8920064-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8920064/providers/microsoft.monitor/accounts/services-j8920064", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-6rhvw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:40:27Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j8920064-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8920064/providers/microsoft.monitor/accounts/services-j8920064", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-6rhvw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:42:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0633472-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.92:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0633472/providers/microsoft.monitor/accounts/services-j0633472", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-kq5jf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:42:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0633472-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.92:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0633472/providers/microsoft.monitor/accounts/services-j0633472", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-79b88f5699-kq5jf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:22:21Z
  Ended:       2026-09-30T17:28:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:22:21Z
  Ended:       2026-09-30T17:28:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="swift-recorder", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="swift-recorder", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet swift-recorder/swift-recorder has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T17:22:21Z
  Ended:       2026-09-30T17:28:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="retina-agent", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/retina-agent has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T17:22:21Z
  Ended:       2026-09-30T17:28:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="set-kubelet-parameters-for-scale", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/services-j5446016", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/set-kubelet-parameters-for-scale has not finished or progressed for at least 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire5
Affected runs (5)
Date (UTC)Associated PRProw job
PR #68202105385002328920064
PR #68202105354569830633472
PR #71782105327789455446016
PR #68202105294563651358720
PR #68202105227914915090432
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
alert44.08%4 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 24: 0 · Sep 25: 2 · Sep 26: 1 · Sep 27: 0 · Sep 28: 5 · Sep 29: 3 · Sep 30: 4none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T19:25:32Z
  Ended:       2026-09-30T19:38:40Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72jx2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T19:25:32Z
  Ended:       2026-09-30T19:38:40Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72jx2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T18:57:01Z
  Ended:       2026-09-30T19:10:58Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72qr2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T18:57:01Z
  Ended:       2026-09-30T19:10:01Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72qr2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 12 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0m156ms43o6rvr33gnk6bjr3pfhe-a6a7t4q2j6v8c3t/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0lqhibjmokvvfkc1lvvbtckpvakc-y8w8x7w3j0c3f3n/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:04:36Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.68:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5446016/providers/microsoft.monitor/accounts/hcps-j5446016", namespace="ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-5hwk9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2t5f0jkpt55pcfu9tg9un71gt3tsm5a6-k9g8q9y9s6m6e2p/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #69702105354536905347072
PR #65802105349698964426752
PR #71782105327789455446016
PR #72612105187789510283264
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e33.06%3 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 14 · Sep 29: 2 · Sep 30: 10none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001666348>: 
    resource group "rg-candidate-5-1-7k2hspfgwlgc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-jdl45b" in resourcegroup="rg-candidate-5-1-7k2hspfgwlgc" to finish deleting: GET https://rp.j2434176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77026e76-9732-4942-8d30-76c6571b6048
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77026e76-9732-4942-8d30-76c6571b6048",
      "name": "77026e76-9732-4942-8d30-76c6571b6048",
      "status": "Failed",
      "startTime": "2026-09-30T19:19:44.434042017Z",
      "endTime": "2026-09-30T19:43:54.082755547Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5gr9ar4ha50b9n6jmkh1qfntpjdb09 still exists (deletion dispatched at 2026-09-30T19:19:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001666348>: 
    resource group "rg-candidate-5-1-7k2hspfgwlgc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-jdl45b" in resourcegroup="rg-candidate-5-1-7k2hspfgwlgc" to finish deleting: GET https://rp.j2434176.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77026e76-9732-4942-8d30-76c6571b6048
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77026e76-9732-4942-8d30-76c6571b6048",
      "name": "77026e76-9732-4942-8d30-76c6571b6048",
      "status": "Failed",
      "startTime": "2026-09-30T19:19:44.434042017Z",
      "endTime": "2026-09-30T19:43:54.082755547Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5gr9ar4ha50b9n6jmkh1qfntpjdb09 still exists (deletion dispatched at 2026-09-30T19:19:46Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002518090>: 
    resource group "node-limit-test-2jbt488tkpl7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-2jbt488tkpl7" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da56d4ee-d492-4bc3-b379-32ba10c1db68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da56d4ee-d492-4bc3-b379-32ba10c1db68",
      "name": "da56d4ee-d492-4bc3-b379-32ba10c1db68",
      "status": "Failed",
      "startTime": "2026-09-30T17:02:22.062008253Z",
      "endTime": "2026-09-30T17:26:25.935485649Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1mqu517if66tcq7vv1883rm6svr2 still exists (deletion dispatched at 2026-09-30T17:02:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc002518090>: 
    resource group "node-limit-test-2jbt488tkpl7": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-2jbt488tkpl7" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da56d4ee-d492-4bc3-b379-32ba10c1db68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/da56d4ee-d492-4bc3-b379-32ba10c1db68",
      "name": "da56d4ee-d492-4bc3-b379-32ba10c1db68",
      "status": "Failed",
      "startTime": "2026-09-30T17:02:22.062008253Z",
      "endTime": "2026-09-30T17:26:25.935485649Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1mqu517if66tcq7vv1883rm6svr2 still exists (deletion dispatched at 2026-09-30T17:02:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001702090>: 
    resource group "fips-enabled-xs9fxbdgnwhz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-xs9fxbdgnwhz" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81d3effe-1da4-43c5-8596-8a54252ca278
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81d3effe-1da4-43c5-8596-8a54252ca278",
      "name": "81d3effe-1da4-43c5-8596-8a54252ca278",
      "status": "Failed",
      "startTime": "2026-09-30T07:45:23.642226584Z",
      "endTime": "2026-09-30T08:09:32.307764104Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56obh88cddik6iio0cu4ci35j02k7f still exists (deletion dispatched at 2026-09-30T07:45:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001702090>: 
    resource group "fips-enabled-xs9fxbdgnwhz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-xs9fxbdgnwhz" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81d3effe-1da4-43c5-8596-8a54252ca278
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/81d3effe-1da4-43c5-8596-8a54252ca278",
      "name": "81d3effe-1da4-43c5-8596-8a54252ca278",
      "status": "Failed",
      "startTime": "2026-09-30T07:45:23.642226584Z",
      "endTime": "2026-09-30T08:09:32.307764104Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56obh88cddik6iio0cu4ci35j02k7f still exists (deletion dispatched at 2026-09-30T07:45:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (10)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #71812105360377792434176
PR #71782105327789455446016
PR #72612105187789510283264
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert33.06%3 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 24: 2 · Sep 25: 12 · Sep 26: 0 · Sep 27: 2 · Sep 28: 7 · Sep 29: 11 · Sep 30: 3none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-01T00:40:12Z
  Ended:       2026-10-01T00:54:14Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7664896-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:53:32Z
  Ended:       2026-09-30T18:09:29Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3795328-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:56:30Z
  Ended:       2026-09-30T18:09:30Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3795328-svc", component="backend", name="clusterclusterserviceupdatedispatch", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterclusterserviceupdatedispatch has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T17:59:28Z
  Ended:       2026-09-30T18:09:30Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3795328-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T15:25:46Z
  Ended:       2026-09-30T15:31:46Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8561152-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #72832105443667987664896
PR #72712105340384333795328
PR #69002105298021888561152
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
alert33.06%3 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsSep 24: 0 · Sep 25: 1 · Sep 26: 1 · Sep 27: 0 · Sep 28: 1 · Sep 29: 2 · Sep 30: 3none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T19:14:32Z
  Ended:       2026-09-30T19:40:31Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="009dd713-cc58-4cfb-b82f-f1fca7b413d6"
  Description: Pod ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-30T19:14:32Z
  Ended:       2026-09-30T19:41:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="009dd713-cc58-4cfb-b82f-f1fca7b413d6"
  Description: Pod ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T18:43:56Z
  Ended:       2026-09-30T19:12:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="bce6bb08-1647-4613-853e-b0a5759423ff"
  Description: Pod ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-30T18:44:57Z
  Ended:       2026-09-30T19:12:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="bce6bb08-1647-4613-853e-b0a5759423ff"
  Description: Pod ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert fired 32 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk", pod="cloud-network-config-controller-6b7699b59d-mtvjg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1aa0eaea-3879-4a75-b74f-75ef531c3b1d"
  Description: Pod ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk/cloud-network-config-controller-6b7699b59d-mtvjg (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o", pod="cloud-network-config-controller-c9f8f4988-98sd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1b3a2719-6e7a-4a5b-aeae-d318e5e4d17e"
  Description: Pod ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o/cloud-network-config-controller-c9f8f4988-98sd7 (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29", pod="csi-snapshot-controller-ff4fdf4ff-8fn8s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="523581cf-5012-43ac-a402-c7a0a8acaac4"
  Description: Pod ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29/csi-snapshot-controller-ff4fdf4ff-8fn8s (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29", pod="cloud-network-config-controller-5c94ccc4b5-cg2cp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="dafd5d14-e303-47e0-b4e1-d01b00ea97f1"
  Description: Pod ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29/cloud-network-config-controller-5c94ccc4b5-cg2cp (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk", pod="cloud-network-config-controller-6b7699b59d-mtvjg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1aa0eaea-3879-4a75-b74f-75ef531c3b1d"
  Description: Pod ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk/cloud-network-config-controller-6b7699b59d-mtvjg (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y", pod="csi-snapshot-controller-7477f9489c-8hwvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1a334069-0f8a-44c9-ab3a-5524390a74c8"
  Description: Pod ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y/csi-snapshot-controller-7477f9489c-8hwvb (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y", pod="cloud-network-config-controller-749db5479f-29nst", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ad67988d-3f68-4e02-84b2-957b828c23ea"
  Description: Pod ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y/cloud-network-config-controller-749db5479f-29nst (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h", pod="cloud-network-config-controller-79d7c84c4-jpmxx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="350fe4f2-fba9-41ad-825a-5166b3b5b8f2"
  Description: Pod ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h/cloud-network-config-controller-79d7c84c4-jpmxx (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o", pod="csi-snapshot-controller-5cc4dd7c9d-2tkdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4766f6e2-d1bc-40e5-9418-ada111a189ef"
  Description: Pod ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o/csi-snapshot-controller-5cc4dd7c9d-2tkdw (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b", pod="csi-snapshot-controller-6445b45cbd-g46qp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4c46f8cc-e7a5-456d-88e9-4d97f220cbc0"
  Description: Pod ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b/csi-snapshot-controller-6445b45cbd-g46qp (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:54:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster", pod="csi-snapshot-controller-68f64f94cf-f2r4v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e4f7cb9e-9e38-408b-b971-4b23a9e6fc0e"
  Description: Pod ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster/csi-snapshot-controller-68f64f94cf-f2r4v (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o", pod="cloud-network-config-controller-c9f8f4988-98sd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1b3a2719-6e7a-4a5b-aeae-d318e5e4d17e"
  Description: Pod ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o/cloud-network-config-controller-c9f8f4988-98sd7 (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b", pod="csi-snapshot-controller-7cf66cc49d-fnzp4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6f42eca0-0eb8-412d-9fbb-e74e80100964"
  Description: Pod ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b/csi-snapshot-controller-7cf66cc49d-fnzp4 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b", pod="csi-snapshot-controller-6445b45cbd-g46qp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4c46f8cc-e7a5-456d-88e9-4d97f220cbc0"
  Description: Pod ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b/csi-snapshot-controller-6445b45cbd-g46qp (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29", pod="csi-snapshot-controller-ff4fdf4ff-8fn8s", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="523581cf-5012-43ac-a402-c7a0a8acaac4"
  Description: Pod ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29/csi-snapshot-controller-ff4fdf4ff-8fn8s (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster", pod="cloud-network-config-controller-85b777dd49-s25s6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="7f4bec03-7c59-4bee-bd1a-b303f9184e5c"
  Description: Pod ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster/cloud-network-config-controller-85b777dd49-s25s6 (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:55:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h", pod="cloud-network-config-controller-79d7c84c4-jpmxx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="350fe4f2-fba9-41ad-825a-5166b3b5b8f2"
  Description: Pod ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h/cloud-network-config-controller-79d7c84c4-jpmxx (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b", pod="csi-snapshot-controller-7cf66cc49d-fnzp4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6f42eca0-0eb8-412d-9fbb-e74e80100964"
  Description: Pod ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b/csi-snapshot-controller-7cf66cc49d-fnzp4 (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster", pod="cloud-network-config-controller-85b777dd49-s25s6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="7f4bec03-7c59-4bee-bd1a-b303f9184e5c"
  Description: Pod ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster/cloud-network-config-controller-85b777dd49-s25s6 (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b", pod="cloud-network-config-controller-c48d57456-gkqfz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="162d1556-afda-4dc1-97da-7279eda02fab"
  Description: Pod ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b/cloud-network-config-controller-c48d57456-gkqfz (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y", pod="cloud-network-config-controller-749db5479f-29nst", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ad67988d-3f68-4e02-84b2-957b828c23ea"
  Description: Pod ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y/cloud-network-config-controller-749db5479f-29nst (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b", pod="cloud-network-config-controller-5687d9df7d-mc789", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c80807d8-f734-41fa-bf72-fa4544a1127f"
  Description: Pod ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b/cloud-network-config-controller-5687d9df7d-mc789 (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h", pod="csi-snapshot-controller-546555d6dd-whrqm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6e800052-3ed4-4832-ac6e-ba3a34c7eda1"
  Description: Pod ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h/csi-snapshot-controller-546555d6dd-whrqm (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk", pod="csi-snapshot-controller-68698d756-2sjcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ea8389c4-6821-4c53-af9f-460b56d932d9"
  Description: Pod ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk/csi-snapshot-controller-68698d756-2sjcb (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b", pod="cloud-network-config-controller-c48d57456-gkqfz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="162d1556-afda-4dc1-97da-7279eda02fab"
  Description: Pod ocm-arohcpci01-2t56ofgpftr9snm78jc1alvisvlihdkq-z9j9b2e5q2j7y1b/cloud-network-config-controller-c48d57456-gkqfz (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o", pod="csi-snapshot-controller-5cc4dd7c9d-2tkdw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="4766f6e2-d1bc-40e5-9418-ada111a189ef"
  Description: Pod ocm-arohcpci01-2t56onftd04005nl721veuiicq8a5s6u-p1f3j3e2l1z4c9o/csi-snapshot-controller-5cc4dd7c9d-2tkdw (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk", pod="csi-snapshot-controller-68698d756-2sjcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ea8389c4-6821-4c53-af9f-460b56d932d9"
  Description: Pod ocm-arohcpci01-2t56ok2d2baqjscldo4vmovim4i18pgh-ephemeral-disk/csi-snapshot-controller-68698d756-2sjcb (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster", pod="csi-snapshot-controller-68f64f94cf-f2r4v", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e4f7cb9e-9e38-408b-b971-4b23a9e6fc0e"
  Description: Pod ocm-arohcpci01-2t56oguqit52ovjjoaakam9v60fven2c-sre-hcp-cluster/csi-snapshot-controller-68f64f94cf-f2r4v (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:56:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h", pod="csi-snapshot-controller-546555d6dd-whrqm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6e800052-3ed4-4832-ac6e-ba3a34c7eda1"
  Description: Pod ocm-arohcpci01-2t56ofblporgmut8brildeurh8bqh0jm-b8a2i8y2x4y0w5h/csi-snapshot-controller-546555d6dd-whrqm (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:57:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y", pod="csi-snapshot-controller-7477f9489c-8hwvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="1a334069-0f8a-44c9-ab3a-5524390a74c8"
  Description: Pod ocm-arohcpci01-2t56obh88cddik6iio0cu4ci35j02k7f-f1s2k3l8b2h6n5y/csi-snapshot-controller-7477f9489c-8hwvb (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:57:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29", pod="cloud-network-config-controller-5c94ccc4b5-cg2cp", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="dafd5d14-e303-47e0-b4e1-d01b00ea97f1"
  Description: Pod ocm-arohcpci01-2t56p42at8ondug5mpb3mmh6f9hh71ck-ea-listrk7c29/cloud-network-config-controller-5c94ccc4b5-cg2cp (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-09-30T08:57:43Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.65.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/hcps-j0283264", namespace="ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b", pod="cloud-network-config-controller-5687d9df7d-mc789", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c80807d8-f734-41fa-bf72-fa4544a1127f"
  Description: Pod ocm-arohcpci01-2t56oclgsj9oapvabnc01b65dqdln9hs-q5s4p4g3c9i1p1b/cloud-network-config-controller-5687d9df7d-mc789 (controller) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #69702105354536905347072
PR #65802105349698964426752
PR #72612105187789510283264
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e22.04%2 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 1 · Sep 28: 0 · Sep 29: 1 · Sep 30: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000cda1a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s9lfl9dtlsjp" to finish creating: GET https://rp.j6843264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000cda1a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-s9lfl9dtlsjp" to finish creating: GET https://rp.j6843264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/165fa122-4b97-461c-a51b-3a9f4e951859"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000b7a960>: 
    failed to create HCP cluster ea-listnpdfsm: failed waiting for cluster="ea-listnpdfsm" in resourcegroup="ea-list-g57slgdp8bj6" to finish creating: GET https://rp.j6843264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000b7a960>: 
    failed to create HCP cluster ea-listnpdfsm: failed waiting for cluster="ea-listnpdfsm" in resourcegroup="ea-list-g57slgdp8bj6" to finish creating: GET https://rp.j6843264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cf1eed55-2ffc-4dee-9cff-f7085dfb10d8"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-7jbkqj" with version 4.21.34 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0011488e0>: 
    failed to create HCP cluster cluster-zstream-4-21-7jbkqj: failed waiting for cluster="cluster-zstream-4-21-7jbkqj" in resourcegroup="rg-zstream-upgrade-4-21-7qqcshqrm588" to finish creating: GET https://rp.j7585920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-7jbkqj" with version 4.21.34 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0011488e0>: 
    failed to create HCP cluster cluster-zstream-4-21-7jbkqj: failed waiting for cluster="cluster-zstream-4-21-7jbkqj" in resourcegroup="rg-zstream-upgrade-4-21-7qqcshqrm588" to finish creating: GET https://rp.j7585920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/30ab22cf-79b3-49a2-b5a2-b420840f6f78"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
Affected runs (2)
Date (UTC)Associated PRProw job
PR #72422105413427336843264
PR #71122105226917077585920
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert22.04%2 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 24: 1 · Sep 25: 3 · Sep 26: 1 · Sep 27: 1 · Sep 28: 1 · Sep 29: 5 · Sep 30: 2none
Full failure examples (2)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T19:11:34Z
  Ended:       2026-09-30T19:36:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72jx2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T19:22:36Z
  Ended:       2026-09-30T19:36:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j5347072-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.33:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5347072/providers/microsoft.monitor/accounts/hcps-j5347072", namespace="ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72jx2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5gnhij1pbtee5r1icg9qlas1lv2h5j-cilium-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T18:52:59Z
  Ended:       2026-09-30T19:08:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72qr2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-30T18:52:59Z
  Ended:       2026-09-30T19:08:58Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j4426752-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.111:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4426752/providers/microsoft.monitor/accounts/hcps-j4426752", namespace="ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-72qr2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t5ga8d53uj3o7nh34nh77qbpeiclu0r-sre-hcp-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #69702105354536905347072
PR #65802105349698964426752
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e11.02%1 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 1 · Sep 27: 0 · Sep 28: 4 · Sep 29: 0 · Sep 30: 6STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001190090>: 
    resource group "rg-zstream-upgrade-5-0-vztfjbhwj86s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-dbc7n2" in resourcegroup="rg-zstream-upgrade-5-0-vztfjbhwj86s" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d21c47-4deb-485c-adc0-8dba2db04e85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d21c47-4deb-485c-adc0-8dba2db04e85",
      "name": "86d21c47-4deb-485c-adc0-8dba2db04e85",
      "status": "Failed",
      "startTime": "2026-09-30T08:00:45.311282131Z",
      "endTime": "2026-09-30T08:24:52.368269792Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ojo9ccai43f073bp4dpipm3b14ma still exists (deletion dispatched at 2026-09-30T08:01:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001190090>: 
    resource group "rg-zstream-upgrade-5-0-vztfjbhwj86s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-dbc7n2" in resourcegroup="rg-zstream-upgrade-5-0-vztfjbhwj86s" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d21c47-4deb-485c-adc0-8dba2db04e85
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d21c47-4deb-485c-adc0-8dba2db04e85",
      "name": "86d21c47-4deb-485c-adc0-8dba2db04e85",
      "status": "Failed",
      "startTime": "2026-09-30T08:00:45.311282131Z",
      "endTime": "2026-09-30T08:24:52.368269792Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ojo9ccai43f073bp4dpipm3b14ma still exists (deletion dispatched at 2026-09-30T08:01:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001344078>: 
    resource group "rg-cp-ystream-upgrade-5-0-fs5gn7c5tqxf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-56tt5z" in resourcegroup="rg-cp-ystream-upgrade-5-0-fs5gn7c5tqxf" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9",
      "name": "52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9",
      "status": "Failed",
      "startTime": "2026-09-30T07:59:56.020744554Z",
      "endTime": "2026-09-30T08:24:02.403347508Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56odktt8bp0otlfbosu69ogttm3nlc still exists (deletion dispatched at 2026-09-30T07:59:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001344078>: 
    resource group "rg-cp-ystream-upgrade-5-0-fs5gn7c5tqxf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-56tt5z" in resourcegroup="rg-cp-ystream-upgrade-5-0-fs5gn7c5tqxf" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9",
      "name": "52d0c5fd-02ac-47eb-a5f2-62e5ca81cff9",
      "status": "Failed",
      "startTime": "2026-09-30T07:59:56.020744554Z",
      "endTime": "2026-09-30T08:24:02.403347508Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56odktt8bp0otlfbosu69ogttm3nlc still exists (deletion dispatched at 2026-09-30T07:59:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e84138>: 
    resource group "rg-zstream-upgrade-4-21-d6bd8lqnx9k6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-mpqc9c" in resourcegroup="rg-zstream-upgrade-4-21-d6bd8lqnx9k6" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9d0d023-161e-4fae-bc07-fa4aac05780f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9d0d023-161e-4fae-bc07-fa4aac05780f",
      "name": "f9d0d023-161e-4fae-bc07-fa4aac05780f",
      "status": "Failed",
      "startTime": "2026-09-30T07:56:48.455291699Z",
      "endTime": "2026-09-30T08:20:52.406066017Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofmkuvbuga9h9ge40bfgpgl5gvfp still exists (deletion dispatched at 2026-09-30T07:57:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000e84138>: 
    resource group "rg-zstream-upgrade-4-21-d6bd8lqnx9k6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-mpqc9c" in resourcegroup="rg-zstream-upgrade-4-21-d6bd8lqnx9k6" to finish deleting: GET https://rp.j0283264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9d0d023-161e-4fae-bc07-fa4aac05780f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f9d0d023-161e-4fae-bc07-fa4aac05780f",
      "name": "f9d0d023-161e-4fae-bc07-fa4aac05780f",
      "status": "Failed",
      "startTime": "2026-09-30T07:56:48.455291699Z",
      "endTime": "2026-09-30T08:20:52.406066017Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56ofmkuvbuga9h9ge40bfgpgl5gvfp still exists (deletion dispatched at 2026-09-30T07:57:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (6)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72612105187789510283264
alert [svc] ServiceCPUDrift fired
full failure pattern:
alert [svc] ServiceCPUDrift fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
alert11.02%1 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 24: 15 · Sep 25: 2 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 2 · Sep 30: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:22:35Z
  Ended:       2026-09-30T17:57:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446016-svc", component="service-memory", container="aro-hcp-frontend", namespace="aro-hcp", pod="aro-hcp-frontend-696bd4cfc8-xrrs2", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-frontend in pod aro-hcp-frontend-696bd4cfc8-xrrs2 (namespace aro-hcp) on cluster ci01-j5446016-svc has averaged 170% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Resolved
  Started:     2026-09-30T17:22:35Z
  Ended:       2026-09-30T18:09:33Z
  Severity:    Sev3
  Labels:      alertname="ServiceCPUDrift", cluster="ci01-j5446016-svc", component="service-memory", container="aro-hcp-frontend", namespace="aro-hcp", pod="aro-hcp-frontend-696bd4cfc8-bzvvh", severity="warning", team="hcp-sl"
  Description: Container aro-hcp-frontend in pod aro-hcp-frontend-696bd4cfc8-bzvvh (namespace aro-hcp) on cluster ci01-j5446016-svc has averaged 209.6% of its CPU request over the last 30 minutes.
This may indicate workload growth that requires right-sizing the request in config.yaml.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert ServiceCPUDrift does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782105327789455446016
alert [svc] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [svc] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 6 days
alert11.02%1 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 6 daysShow trend detailsSep 24: 1 · Sep 25: 1 · Sep 26: 1 · Sep 27: 0 · Sep 28: 2 · Sep 29: 2 · Sep 30: 1none
Full failure examples (1)
alert fired 12 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T06:32:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T06:32:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T06:39:00Z
  Ended:       2026-09-30T07:01:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55he6nad38adkphki745bqugvhhgmi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55he6nad38adkphki745bqugvhhgmi/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T06:39:00Z
  Ended:       2026-09-30T07:01:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55he6nad38adkphki745bqugvhhgmi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55he6nad38adkphki745bqugvhhgmi/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-30T07:02:01Z
  Ended:       2026-09-30T07:08:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55s5ip2mfud3dnigkagbgr229suldb", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55s5ip2mfud3dnigkagbgr229suldb/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-30T07:02:01Z
  Ended:       2026-09-30T07:08:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55sahljftd8u6q1m4sbae73q1qo4f2", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55sahljftd8u6q1m4sbae73q1qo4f2/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-30T07:02:01Z
  Ended:       2026-09-30T07:08:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55s5ip2mfud3dnigkagbgr229suldb", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55s5ip2mfud3dnigkagbgr229suldb/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-30T07:02:01Z
  Ended:       2026-09-30T07:08:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55sahljftd8u6q1m4sbae73q1qo4f2", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55sahljftd8u6q1m4sbae73q1qo4f2/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-30T07:03:02Z
  Ended:       2026-09-30T07:12:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55soh6c16uoks7fn5rqknhr3g6cc88", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55soh6c16uoks7fn5rqknhr3g6cc88/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-30T07:03:02Z
  Ended:       2026-09-30T07:12:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55soh6c16uoks7fn5rqknhr3g6cc88", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55soh6c16uoks7fn5rqknhr3g6cc88/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-30T07:03:02Z
  Ended:       2026-09-30T07:11:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55som67oi1sqntaq9acgv9gavh7vjf", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55som67oi1sqntaq9acgv9gavh7vjf/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-30T07:03:02Z
  Ended:       2026-09-30T07:11:06Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55som67oi1sqntaq9acgv9gavh7vjf", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t55som67oi1sqntaq9acgv9gavh7vjf/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71582105165316576251904
alert [svc] KubeDeploymentRolloutStuck fired
full failure pattern:
alert [svc] KubeDeploymentRolloutStuck fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
alert11.02%1 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsSep 24: 1 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 2 · Sep 30: 1none
Full failure examples (1)
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T06:42:08Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T06:42:08Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-30T06:49:03Z
  Ended:       2026-09-30T07:01:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55he6nad38adkphki745bqugvhhgmi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2t55he6nad38adkphki745bqugvhhgmi/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-30T06:49:04Z
  Ended:       2026-09-30T07:01:02Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6251904-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.183:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6251904/providers/microsoft.monitor/accounts/services-j6251904", namespace="klusterlet-2t55he6nad38adkphki745bqugvhhgmi", pod="arohcp-monitor-kube-state-metrics-7988dc648c-blwss", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment klusterlet-2t55he6nad38adkphki745bqugvhhgmi/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDeploymentRolloutStuck does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71582105165316576251904
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial...
full failure pattern:
VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e11.02%1 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 1 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000e5b548>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
    <*errors.joinError | 0xc000e5b548>: 
    VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-46kd7.apps.aro.cilium-cl.f3k6.j5347072.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69702105354536905347072
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days
e2e11.02%1 of 98 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 3 daysShow trend detailsSep 24: 0 · Sep 25: 1 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc000c9ee28>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation
Unexpected error:
    <*errors.joinError | 0xc000c9ee28>: 
    context deadline exceeded
    failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69442105295109229645824
--------------------------------------------------------------------------------
full failure pattern:
--------------------------------------------------------------------------------
Signal: NoiseSignal: Noise — mostly punctuation
e2e22.04%2 of 98 job runs affectedNoiseSignal: Noise — mostly punctuationShow trend detailsSep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-9xv2vk78f5m7" to finish creating: GET https://rp.j4339200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f34fa79-f2bb-47f4-8491-50e2c8ff8e21
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f34fa79-f2bb-47f4-8491-50e2c8ff8e21",
  "name": "9f34fa79-f2bb-47f4-8491-50e2c8ff8e21",
  "status": "Failed",
  "startTime": "2026-09-30T22:15:27.259108656Z",
  "endTime": "2026-09-30T22:34:33.885780975Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-9xv2vk78f5m7--managed/providers/Microsoft.Compute/virtualMachines/v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-9xv2vk78f5m7--managed/providers/Microsoft.Compute/virtualMachines/v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79 to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-9xv2vk78f5m7" to finish creating: GET https://rp.j4339200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f34fa79-f2bb-47f4-8491-50e2c8ff8e21
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9f34fa79-f2bb-47f4-8491-50e2c8ff8e21",
  "name": "9f34fa79-f2bb-47f4-8491-50e2c8ff8e21",
  "status": "Failed",
  "startTime": "2026-09-30T22:15:27.259108656Z",
  "endTime": "2026-09-30T22:34:33.885780975Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-9xv2vk78f5m7--managed/providers/Microsoft.Compute/virtualMachines/v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-9xv2vk78f5m7--managed/providers/Microsoft.Compute/virtualMachines/v7o2l4s3b0x1k3m-np-update-nodes-z48kk-nvb79 to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-78dt7bb9qdx9" to finish creating: GET https://rp.j9082368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f34a8c-014f-4784-8b2d-d242f6bfcb85
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f34a8c-014f-4784-8b2d-d242f6bfcb85",
  "name": "e3f34a8c-014f-4784-8b2d-d242f6bfcb85",
  "status": "Failed",
  "startTime": "2026-09-30T05:49:26.691969203Z",
  "endTime": "2026-09-30T06:08:29.964580175Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-78dt7bb9qdx9--managed/providers/Microsoft.Compute/virtualMachines/t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-78dt7bb9qdx9--managed/providers/Microsoft.Compute/virtualMachines/t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-78dt7bb9qdx9" to finish creating: GET https://rp.j9082368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f34a8c-014f-4784-8b2d-d242f6bfcb85
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3f34a8c-014f-4784-8b2d-d242f6bfcb85",
  "name": "e3f34a8c-014f-4784-8b2d-d242f6bfcb85",
  "status": "Failed",
  "startTime": "2026-09-30T05:49:26.691969203Z",
  "endTime": "2026-09-30T06:08:29.964580175Z",
  "error": {
    "code": "DeadlineExceeded",
    "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-78dt7bb9qdx9--managed/providers/Microsoft.Compute/virtualMachines/t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-update-nodes-78dt7bb9qdx9--managed/providers/Microsoft.Compute/virtualMachines/t1n4p5j2m9y1n6b-np-update-nodes-fvh6z-s276d to exist\n"
  }
}
--------------------------------------------------------------------------------
]
Expected
    <[]error | len:1, cap:1>: ...
to be empty
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #72792105400327724339200
PR #72292105159944369082368
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)
e2e88.16%8 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)Show trend detailsSep 24: 0 · Sep 25: 6 · Sep 26: 2 · Sep 27: 7 · Sep 28: 19 · Sep 29: 10 · Sep 30: 8INT, PROD, STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc000ca3800>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-xw6tpr5d2lsl" to finish creating: GET https://rp.j4862592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc",
      "name": "3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc",
      "status": "Failed",
      "startTime": "2026-09-30T17:03:28.253892004Z",
      "endTime": "2026-09-30T17:22:36.043787859Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-h2g79-bb4np: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xw6tpr5d2lsl--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-h2g79-bb4np to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-h2g79-bb4np: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xw6tpr5d2lsl--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-h2g79-bb4np to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc000ca3800>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-xw6tpr5d2lsl" to finish creating: GET https://rp.j4862592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc",
      "name": "3d2602fc-b3fa-43e1-b8e3-5be7f209f5fc",
      "status": "Failed",
      "startTime": "2026-09-30T17:03:28.253892004Z",
      "endTime": "2026-09-30T17:22:36.043787859Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine oidc-wi-cluster-oidc-wi-np-h2g79-bb4np: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xw6tpr5d2lsl--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-h2g79-bb4np to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine oidc-wi-cluster-oidc-wi-np-h2g79-bb4np: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-xw6tpr5d2lsl--managed/providers/Microsoft.Compute/virtualMachines/oidc-wi-cluster-oidc-wi-np-h2g79-bb4np to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000686fa0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-hqw2kg" in resourcegroup="rg-np-downgrade-hqw2kg-tzshqkp8zjqk" to finish creating: GET https://rp.j0694272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc",
      "name": "0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc",
      "status": "Failed",
      "startTime": "2026-09-30T14:18:31.838214826Z",
      "endTime": "2026-09-30T14:37:32.428641421Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-hqw2kg-tzshqkp8zjqk-np-dg-hqw2kg--managed/providers/Microsoft.Compute/virtualMachines/h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-hqw2kg-tzshqkp8zjqk-np-dg-hqw2kg--managed/providers/Microsoft.Compute/virtualMachines/h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:600]: failed to create nodepool npdg-4-21
Unexpected error:
    <*fmt.wrapError | 0xc000686fa0>: 
    failed to create NodePool npdg-4-21: failed waiting for nodepool="npdg-4-21" for cluster "np-downgrade-hqw2kg" in resourcegroup="rg-np-downgrade-hqw2kg-tzshqkp8zjqk" to finish creating: GET https://rp.j0694272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc",
      "name": "0c88c27e-4a0d-4fb4-a4af-b21df9d3ccdc",
      "status": "Failed",
      "startTime": "2026-09-30T14:18:31.838214826Z",
      "endTime": "2026-09-30T14:37:32.428641421Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-hqw2kg-tzshqkp8zjqk-np-dg-hqw2kg--managed/providers/Microsoft.Compute/virtualMachines/h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-hqw2kg-tzshqkp8zjqk-np-dg-hqw2kg--managed/providers/Microsoft.Compute/virtualMachines/h2z3j3d0p8e5c3v-npdg-4-21-7x86w-jpqjj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wsv85m" for cluster "cluster-candidate-4-23-wsv85m"
Unexpected error:
    <*fmt.wrapError | 0xc000f7d6c0>: 
    failed to create NodePool np-wsv85m: failed waiting for nodepool="np-wsv85m" for cluster "cluster-candidate-4-23-wsv85m" in resourcegroup="rg-candidate-4-23-x5s4hpmkwzhp" to finish creating: GET https://rp.j1071872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/edf97b61-1366-48f5-936f-5378dbccb550
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/edf97b61-1366-48f5-936f-5378dbccb550",
      "name": "edf97b61-1366-48f5-936f-5378dbccb550",
      "status": "Failed",
      "startTime": "2026-09-30T11:35:07.81865953Z",
      "endTime": "2026-09-30T11:54:13.175354213Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-x5s4hpmkwzhp-candidate-wsv85m--managed/providers/Microsoft.Compute/virtualMachines/v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-x5s4hpmkwzhp-candidate-wsv85m--managed/providers/Microsoft.Compute/virtualMachines/v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wsv85m" for cluster "cluster-candidate-4-23-wsv85m"
Unexpected error:
    <*fmt.wrapError | 0xc000f7d6c0>: 
    failed to create NodePool np-wsv85m: failed waiting for nodepool="np-wsv85m" for cluster "cluster-candidate-4-23-wsv85m" in resourcegroup="rg-candidate-4-23-x5s4hpmkwzhp" to finish creating: GET https://rp.j1071872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/edf97b61-1366-48f5-936f-5378dbccb550
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/edf97b61-1366-48f5-936f-5378dbccb550",
      "name": "edf97b61-1366-48f5-936f-5378dbccb550",
      "status": "Failed",
      "startTime": "2026-09-30T11:35:07.81865953Z",
      "endTime": "2026-09-30T11:54:13.175354213Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-x5s4hpmkwzhp-candidate-wsv85m--managed/providers/Microsoft.Compute/virtualMachines/v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-x5s4hpmkwzhp-candidate-wsv85m--managed/providers/Microsoft.Compute/virtualMachines/v9b2q3h5h4o9b5j-np-wsv85m-qtz79-w22nd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (8)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (8)
Date (UTC)Associated PRProw job
PR #72422105320779594862592
PR #70532105279807200694272
PR #69002105244373301071872
PR #66622105229330492690432
PR #69152105225367915597824
PR #72622105203604242567168
PR #68842105193266038181888
PR #69702105132902411931648
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e44.08%4 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsSep 24: 0 · Sep 25: 3 · Sep 26: 0 · Sep 27: 0 · Sep 28: 7 · Sep 29: 10 · Sep 30: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-z8sw852rvcl6/cluster-candidate-5-1-f8bld9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f24640>: 
    failed to create HCP cluster cluster-candidate-5-1-f8bld9: failed waiting for cluster="cluster-candidate-5-1-f8bld9" in resourcegroup="rg-candidate-5-1-z8sw852rvcl6" to finish creating: GET https://rp.j5298560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1da1b003-1c66-47c8-99b2-f49235ea2736
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1da1b003-1c66-47c8-99b2-f49235ea2736",
      "name": "1da1b003-1c66-47c8-99b2-f49235ea2736",
      "status": "Failed",
      "startTime": "2026-09-30T17:01:56.141378012Z",
      "endTime": "2026-09-30T17:30:57.762149233Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 24m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-z8sw852rvcl6/cluster-candidate-5-1-f8bld9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f24640>: 
    failed to create HCP cluster cluster-candidate-5-1-f8bld9: failed waiting for cluster="cluster-candidate-5-1-f8bld9" in resourcegroup="rg-candidate-5-1-z8sw852rvcl6" to finish creating: GET https://rp.j5298560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1da1b003-1c66-47c8-99b2-f49235ea2736
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1da1b003-1c66-47c8-99b2-f49235ea2736",
      "name": "1da1b003-1c66-47c8-99b2-f49235ea2736",
      "status": "Failed",
      "startTime": "2026-09-30T17:01:56.141378012Z",
      "endTime": "2026-09-30T17:30:57.762149233Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 24m10s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-622nv7gb94zh/cluster-candidate-4-22-jcthkg should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c422a0>: 
    failed to create HCP cluster cluster-candidate-4-22-jcthkg: failed waiting for cluster="cluster-candidate-4-22-jcthkg" in resourcegroup="rg-candidate-4-22-622nv7gb94zh" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9",
      "name": "6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:38.429049015Z",
      "endTime": "2026-09-30T17:21:45.984146934Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 22m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-622nv7gb94zh/cluster-candidate-4-22-jcthkg should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c422a0>: 
    failed to create HCP cluster cluster-candidate-4-22-jcthkg: failed waiting for cluster="cluster-candidate-4-22-jcthkg" in resourcegroup="rg-candidate-4-22-622nv7gb94zh" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9",
      "name": "6c7ea5b6-e38d-4282-a8bd-b4e196c0a2b9",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:38.429049015Z",
      "endTime": "2026-09-30T17:21:45.984146934Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.16 is Partial (want Completed), started 22m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-9z27ks86nc27/cluster-candidate-5-1-hrwtgg should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010a04c0>: 
    failed to create HCP cluster cluster-candidate-5-1-hrwtgg: failed waiting for cluster="cluster-candidate-5-1-hrwtgg" in resourcegroup="rg-candidate-5-1-9z27ks86nc27" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9135939f-e992-427f-be1e-4ecfae2b319a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9135939f-e992-427f-be1e-4ecfae2b319a",
      "name": "9135939f-e992-427f-be1e-4ecfae2b319a",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:29.681806994Z",
      "endTime": "2026-09-30T17:21:35.77427567Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 22m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-9z27ks86nc27/cluster-candidate-5-1-hrwtgg should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010a04c0>: 
    failed to create HCP cluster cluster-candidate-5-1-hrwtgg: failed waiting for cluster="cluster-candidate-5-1-hrwtgg" in resourcegroup="rg-candidate-5-1-9z27ks86nc27" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9135939f-e992-427f-be1e-4ecfae2b319a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9135939f-e992-427f-be1e-4ecfae2b319a",
      "name": "9135939f-e992-427f-be1e-4ecfae2b319a",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:29.681806994Z",
      "endTime": "2026-09-30T17:21:35.77427567Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-ec.1 is Partial (want Completed), started 22m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.12
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #71242105331328655298560
PR #71782105327789455446016
PR #72612105310240105828352
PR #71412105297136064139264
alert [svc] ArobitForwarderJobUp fired
full failure pattern:
alert [svc] ArobitForwarderJobUp fired
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
alert44.08%4 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsSep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 4none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T20:39:32Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j8920064-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j8920064-svc has been unreachable for 15 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T18:42:29Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j0633472-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j0633472-svc has been unreachable for 15 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T14:51:34Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j1358720-svc", component="monitoring-infrastructure", region="canadacentral", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j1358720-svc has been unreachable for 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert ArobitForwarderJobUp does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #68202105385002328920064
PR #68202105354569830633472
PR #68202105294563651358720
PR #68202105227914915090432
failed to roll out Helm release: context deadline exceeded
full failure pattern:
failed to roll out Helm release: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
provision44.08%4 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsSep 24: 1 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 4none
Full failure examples (3)
time=2026-09-30T20:23:20.071Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T20:23:20.785Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j8920064-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j8920064-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T20:23:20.785Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:23:21.014Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T20:23:21.176Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T20:23:21.176Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:23:21.232Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:33:24.255Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T20:33:24.255Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T20:34:24.257Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T20:34:24.848Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j8920064-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j8920064-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T20:34:24.848Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:34:25.188Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T20:34:25.588Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:34:25.588Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:34:25.757Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T20:34:26.389Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:34:26.390Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:34:26.616Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:34:26.899Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T20:34:27.340Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T20:34:27.673Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:34:27.873Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:34:28.054Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:34:28.402Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T20:34:28.402Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:34:28.487Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T20:44:30.230Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T20:44:30.230Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T20:45:30.230Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T20:45:30.710Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j8920064-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j8920064-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#87a67f37-4acf-40f9-a2aa-33f7bdfa8ae1 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T20:45:30.711Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:45:31.093Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T20:45:31.467Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:45:31.467Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:45:31.591Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T20:45:32.324Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:45:32.324Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:45:32.528Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:45:32.928Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T20:45:33.424Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T20:45:33.656Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:45:34.121Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:45:34.518Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T20:45:34.878Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T20:45:34.878Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T20:45:34.950Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T20:53:20.192Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-30T20:53:20.193Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"
time=2026-09-30T18:26:03.649Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T18:26:04.095Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j0633472-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j0633472-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:42f17d2a-2d54-4fa2-b94e-06bedcfbde0d region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#42f17d2a-2d54-4fa2-b94e-06bedcfbde0d INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T18:26:04.095Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:26:04.389Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T18:26:04.602Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T18:26:04.602Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:26:04.633Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:36:07.312Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T18:36:07.312Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T18:37:07.314Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T18:37:08.026Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j0633472-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j0633472-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:42f17d2a-2d54-4fa2-b94e-06bedcfbde0d region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#42f17d2a-2d54-4fa2-b94e-06bedcfbde0d INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T18:37:08.026Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:37:08.359Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T18:37:08.708Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:37:08.708Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:37:08.746Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T18:37:09.698Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:37:09.698Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:37:09.874Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:37:10.131Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T18:37:10.610Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T18:37:10.790Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:37:11.050Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:37:11.400Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:37:11.717Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T18:37:11.717Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:37:11.766Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T18:47:13.248Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T18:47:13.248Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T18:48:13.252Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T18:48:13.967Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j0633472-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j0633472-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:42f17d2a-2d54-4fa2-b94e-06bedcfbde0d region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#42f17d2a-2d54-4fa2-b94e-06bedcfbde0d INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T18:48:13.968Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:48:14.374Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T18:48:14.654Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:48:14.654Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:48:14.778Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T18:48:15.694Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:48:15.694Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:48:16.096Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:48:16.376Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T18:48:16.694Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T18:48:16.931Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:48:17.265Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:48:17.464Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T18:48:17.811Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T18:48:17.811Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T18:48:17.876Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T18:56:03.654Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-30T18:56:03.655Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"
time=2026-09-30T14:35:13.661Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T14:35:14.082Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j1358720-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j1358720-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e962c7b1-5eb5-497b-9341-4b3a121f354e region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#e962c7b1-5eb5-497b-9341-4b3a121f354e INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T14:35:14.083Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:35:14.275Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T14:35:14.491Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T14:35:14.491Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:35:14.524Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:45:17.368Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T14:45:17.369Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T14:46:17.370Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T14:46:17.815Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j1358720-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j1358720-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e962c7b1-5eb5-497b-9341-4b3a121f354e region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#e962c7b1-5eb5-497b-9341-4b3a121f354e INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T14:46:17.815Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:46:18.042Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T14:46:18.350Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:46:18.350Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:46:18.601Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T14:46:19.301Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:46:19.301Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:46:19.487Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:46:19.912Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T14:46:20.506Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T14:46:20.869Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:46:21.189Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:46:21.550Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:46:21.905Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T14:46:21.905Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:46:21.951Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-30T14:56:23.257Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T14:56:23.257Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/3\ncontext deadline exceeded"
time=2026-09-30T14:57:23.259Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-30T14:57:24.108Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j1358720-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:eb3e2e068172121a9ca133c5d17784df770dacc104f20c7d4a7df5f1c70e8a08 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:450m memory:472Mi]]] geoShortId:ca kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:canadacentral role:hcp-underlay-ci01-j1358720-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5c29681bb4b38d08ca21c31c4553bab72e8e9c260dc0ec875e1d014268a9e580 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:e962c7b1-5eb5-497b-9341-4b3a121f354e region:canadacentral secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#e962c7b1-5eb5-497b-9341-4b3a121f354e INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-30T14:57:24.108Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:57:24.411Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-30T14:57:24.749Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:57:24.749Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:57:25.002Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T14:57:25.620Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:57:25.620Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:57:25.977Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:57:26.226Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-30T14:57:26.765Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-30T14:57:26.974Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:57:27.201Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:57:27.763Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-30T14:57:28.017Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-30T14:57:28.017Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-30T14:57:28.079Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-30T15:05:13.695Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-30T15:05:13.695Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/arobit4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #68202105385002328920064
PR #68202105354569830633472
PR #68202105294563651358720
PR #68202105227914915090432
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)
e2e33.06%3 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 6 day(s)Show trend detailsSep 24: 0 · Sep 25: 2 · Sep 26: 1 · Sep 27: 8 · Sep 28: 1 · Sep 29: 19 · Sep 30: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001128760>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-h59xkph9dfpm" to finish creating: GET https://rp.j5347072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88da31a3-069e-4240-80a2-676685212f2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88da31a3-069e-4240-80a2-676685212f2c",
      "name": "88da31a3-069e-4240-80a2-676685212f2c",
      "status": "Failed",
      "startTime": "2026-09-30T18:49:41.355729108Z",
      "endTime": "2026-09-30T19:18:42.930044888Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001128760>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-h59xkph9dfpm" to finish creating: GET https://rp.j5347072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88da31a3-069e-4240-80a2-676685212f2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88da31a3-069e-4240-80a2-676685212f2c",
      "name": "88da31a3-069e-4240-80a2-676685212f2c",
      "status": "Failed",
      "startTime": "2026-09-30T18:49:41.355729108Z",
      "endTime": "2026-09-30T19:18:42.930044888Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00120c640>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-mnp9svrwqcpp" to finish creating: GET https://rp.j4426752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fd78a0f-65d2-4ffb-8dde-60a82af78dd0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fd78a0f-65d2-4ffb-8dde-60a82af78dd0",
      "name": "9fd78a0f-65d2-4ffb-8dde-60a82af78dd0",
      "status": "Failed",
      "startTime": "2026-09-30T18:21:17.653151468Z",
      "endTime": "2026-09-30T18:50:22.965863416Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00120c640>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-mnp9svrwqcpp" to finish creating: GET https://rp.j4426752.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fd78a0f-65d2-4ffb-8dde-60a82af78dd0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9fd78a0f-65d2-4ffb-8dde-60a82af78dd0",
      "name": "9fd78a0f-65d2-4ffb-8dde-60a82af78dd0",
      "status": "Failed",
      "startTime": "2026-09-30T18:21:17.653151468Z",
      "endTime": "2026-09-30T18:50:22.965863416Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-m98nfkv66wgc/cluster-candidate-4-23-2zm5lz should provision
Unexpected error:
    <*fmt.wrapError | 0xc0013ea120>: 
    failed to create HCP cluster cluster-candidate-4-23-2zm5lz: failed waiting for cluster="cluster-candidate-4-23-2zm5lz" in resourcegroup="rg-candidate-4-23-m98nfkv66wgc" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e5bac24-0540-4cfd-8079-7530283faa59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e5bac24-0540-4cfd-8079-7530283faa59",
      "name": "7e5bac24-0540-4cfd-8079-7530283faa59",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:39.678276861Z",
      "endTime": "2026-09-30T17:21:45.946479348Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-m98nfkv66wgc/cluster-candidate-4-23-2zm5lz should provision
Unexpected error:
    <*fmt.wrapError | 0xc0013ea120>: 
    failed to create HCP cluster cluster-candidate-4-23-2zm5lz: failed waiting for cluster="cluster-candidate-4-23-2zm5lz" in resourcegroup="rg-candidate-4-23-m98nfkv66wgc" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e5bac24-0540-4cfd-8079-7530283faa59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7e5bac24-0540-4cfd-8079-7530283faa59",
      "name": "7e5bac24-0540-4cfd-8079-7530283faa59",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:39.678276861Z",
      "endTime": "2026-09-30T17:21:45.946479348Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #69702105354536905347072
PR #65802105349698964426752
PR #71782105327789455446016
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e22.04%2 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 29 · Sep 29: 17 · Sep 30: 22none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00113a0c0>: 
    resource group "private-kas-hm9pdnmdszt5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-hm9pdnmdszt5" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f588ad3-c1af-45bb-9548-096867a64bbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f588ad3-c1af-45bb-9548-096867a64bbf",
      "name": "2f588ad3-c1af-45bb-9548-096867a64bbf",
      "status": "Failed",
      "startTime": "2026-09-30T17:29:27.633769413Z",
      "endTime": "2026-09-30T17:53:32.876325702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0n6rovpqtd5op2d9vogdgft7tah1 still exists (deletion dispatched at 2026-09-30T17:29:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00113a0c0>: 
    resource group "private-kas-hm9pdnmdszt5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-hm9pdnmdszt5" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f588ad3-c1af-45bb-9548-096867a64bbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2f588ad3-c1af-45bb-9548-096867a64bbf",
      "name": "2f588ad3-c1af-45bb-9548-096867a64bbf",
      "status": "Failed",
      "startTime": "2026-09-30T17:29:27.633769413Z",
      "endTime": "2026-09-30T17:53:32.876325702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f0n6rovpqtd5op2d9vogdgft7tah1 still exists (deletion dispatched at 2026-09-30T17:29:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0029c4180>: 
    resource group "feature-aggregation-bffzrcs9xcvk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-bffzrcs9xcvk" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2db4f57-9d20-454f-8d38-b79b992f2f32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2db4f57-9d20-454f-8d38-b79b992f2f32",
      "name": "b2db4f57-9d20-454f-8d38-b79b992f2f32",
      "status": "Failed",
      "startTime": "2026-09-30T17:34:12.215869805Z",
      "endTime": "2026-09-30T17:58:12.821045662Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1rm3k8s9324kfkruicq09u1qjvn0 still exists (deletion dispatched at 2026-09-30T17:34:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0029c4180>: 
    resource group "feature-aggregation-bffzrcs9xcvk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-bffzrcs9xcvk" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2db4f57-9d20-454f-8d38-b79b992f2f32
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b2db4f57-9d20-454f-8d38-b79b992f2f32",
      "name": "b2db4f57-9d20-454f-8d38-b79b992f2f32",
      "status": "Failed",
      "startTime": "2026-09-30T17:34:12.215869805Z",
      "endTime": "2026-09-30T17:58:12.821045662Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1rm3k8s9324kfkruicq09u1qjvn0 still exists (deletion dispatched at 2026-09-30T17:34:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010c0210>: 
    resource group "private-keyvault-9g9ksdvdr2w9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-9g9ksdvdr2w9" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/135095ef-4130-48de-a890-36a3f2ec4ba2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/135095ef-4130-48de-a890-36a3f2ec4ba2",
      "name": "135095ef-4130-48de-a890-36a3f2ec4ba2",
      "status": "Failed",
      "startTime": "2026-09-30T17:18:16.6672625Z",
      "endTime": "2026-09-30T17:42:26.002126676Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1cjpvuq75dcn2gn08v3gqss4ksup still exists (deletion dispatched at 2026-09-30T17:18:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0010c0210>: 
    resource group "private-keyvault-9g9ksdvdr2w9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-9g9ksdvdr2w9" to finish deleting: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/135095ef-4130-48de-a890-36a3f2ec4ba2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/135095ef-4130-48de-a890-36a3f2ec4ba2",
      "name": "135095ef-4130-48de-a890-36a3f2ec4ba2",
      "status": "Failed",
      "startTime": "2026-09-30T17:18:16.6672625Z",
      "endTime": "2026-09-30T17:42:26.002126676Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5f1cjpvuq75dcn2gn08v3gqss4ksup still exists (deletion dispatched at 2026-09-30T17:18:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (18)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71782105327789455446016
PR #72612105187789510283264
alert [svc] KustoLogsDataStale fired
full failure pattern:
alert [svc] KustoLogsDataStale fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
alert22.04%2 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1 · Sep 29: 2 · Sep 30: 2none
Full failure examples (2)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T12:05:05Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j9346176-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.63:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9346176/providers/microsoft.monitor/accounts/services-j9346176", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c94b5c67c-s5vnr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j9346176-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-30T12:05:05Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j9346176-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.63:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9346176/providers/microsoft.monitor/accounts/services-j9346176", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-c94b5c67c-s5vnr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="aro-hcp-exporter", severity="warning", table="systemdlogs"
  Description: Kusto log data for table systemdlogs on cluster ci01-j9346176-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T09:05:33Z
  Ended:       2026-09-30T09:25:32Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0283264-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.44:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/services-j0283264", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-8bc58bd6b-kmgbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="aro-hcp-exporter", severity="warning", table="kubernetesevents"
  Description: Kusto log data for table kubernetesevents on cluster ci01-j0283264-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.

Firing 2:
  State:       Resolved
  Started:     2026-09-30T09:06:32Z
  Ended:       2026-09-30T09:26:32Z
  Severity:    Sev3
  Labels:      alertname="KustoLogsDataStale", cluster="ci01-j0283264-svc", component="kusto", container="aro-hcp-exporter", endpoint="metrics", environment="ci01", instance="10.128.64.44:8080", job="aro-hcp-exporter", kusto_cluster="hcp-dev-us-2", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0283264/providers/microsoft.monitor/accounts/services-j0283264", namespace="aro-hcp-exporter", pod="aro-hcp-exporter-8bc58bd6b-kmgbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="aro-hcp-exporter", severity="warning", table="kubernetesevents"
  Description: Kusto log data for table kubernetesevents on cluster ci01-j0283264-svc (Kusto cluster hcp-dev-us-2) is stale. Check the ingestion pipeline.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KustoLogsDataStale does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #68842105230891239346176
PR #72612105187789510283264
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e22.04%2 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 2STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003c2f500>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4t7wr4mxhmz6" to finish creating: GET https://rp.j2153728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/911cbd71-d1cd-4d26-a932-ae7e6da8fb9f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/911cbd71-d1cd-4d26-a932-ae7e6da8fb9f",
      "name": "911cbd71-d1cd-4d26-a932-ae7e6da8fb9f",
      "status": "Failed",
      "startTime": "2026-09-30T23:49:39.505681667Z",
      "endTime": "2026-10-01T00:08:45.705788813Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4t7wr4mxhmz6--managed/providers/Microsoft.Compute/virtualMachines/u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4t7wr4mxhmz6--managed/providers/Microsoft.Compute/virtualMachines/u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc003c2f500>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4t7wr4mxhmz6" to finish creating: GET https://rp.j2153728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/911cbd71-d1cd-4d26-a932-ae7e6da8fb9f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/911cbd71-d1cd-4d26-a932-ae7e6da8fb9f",
      "name": "911cbd71-d1cd-4d26-a932-ae7e6da8fb9f",
      "status": "Failed",
      "startTime": "2026-09-30T23:49:39.505681667Z",
      "endTime": "2026-10-01T00:08:45.705788813Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4t7wr4mxhmz6--managed/providers/Microsoft.Compute/virtualMachines/u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4t7wr4mxhmz6--managed/providers/Microsoft.Compute/virtualMachines/u3l1m0a6z5n1f9b-np-deletable-265vx-bw7dl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000c97540>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4vhtkz65lcdf" to finish creating: GET https://rp.j6898944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f923a9cc-3bf2-46c5-a14f-9498c8c16cec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f923a9cc-3bf2-46c5-a14f-9498c8c16cec",
      "name": "f923a9cc-3bf2-46c5-a14f-9498c8c16cec",
      "status": "Failed",
      "startTime": "2026-09-30T07:58:35.462676175Z",
      "endTime": "2026-09-30T08:17:44.959895115Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4vhtkz65lcdf--managed/providers/Microsoft.Compute/virtualMachines/v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4vhtkz65lcdf--managed/providers/Microsoft.Compute/virtualMachines/v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000c97540>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4vhtkz65lcdf" to finish creating: GET https://rp.j6898944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f923a9cc-3bf2-46c5-a14f-9498c8c16cec
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f923a9cc-3bf2-46c5-a14f-9498c8c16cec",
      "name": "f923a9cc-3bf2-46c5-a14f-9498c8c16cec",
      "status": "Failed",
      "startTime": "2026-09-30T07:58:35.462676175Z",
      "endTime": "2026-09-30T08:17:44.959895115Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4vhtkz65lcdf--managed/providers/Microsoft.Compute/virtualMachines/v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4vhtkz65lcdf--managed/providers/Microsoft.Compute/virtualMachines/v6y0n4l6q1w8i3p-np-delete-main-p2bwr-dv274 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #72722105426318702153728
PR #72602105189267666898944
alert [hcp] HCPClusterOperatorUnavailable fired
full failure pattern:
alert [hcp] HCPClusterOperatorUnavailable fired
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
alert22.04%2 of 98 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 2none
Full failure examples (2)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T17:40:28Z
  Ended:       2026-09-30T18:18:32Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j5446016-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t5f0hjlm2j48ooq92uj0su5rbnsq03t-x3y7i6e5z1z9c0c", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t5f0hjlm2j48ooq92uj0su5rbnsq03t-x3y7i6e5z1z9c0c (management cluster ci01-j5446016-mgmt-1) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-30T09:10:27Z
  Ended:       2026-09-30T10:12:28Z
  Severity:    Sev3
  Labels:      alertname="HCPClusterOperatorUnavailable", cluster="ci01-j0283264-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t56ofmlsp87gpoiuva34g4ukacv58an-private-kas", region="westus3", severity="warning"
  Description: 2 cluster operator(s) on hosted cluster ocm-arohcpci01-2t56ofmlsp87gpoiuva34g4ukacv58an-private-kas (management cluster ci01-j0283264-mgmt-2) have been reporting Available=false for more than 30 minutes. The version and console operators are excluded from this alert; the affected cluster has worker nodes present. An unavailable operator means the component it manages is down, not merely degraded.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert HCPClusterOperatorUnavailable does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71782105327789455446016
PR #72612105187789510283264
expected node pool nodes to report at least one release image ref before upgrade
full failure pattern:
expected node pool nodes to report at least one release image ref before upgrade
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2 · Sep 29: 0 · Sep 30: 3none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:180]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:180]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:476]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:476]: expected node pool nodes to report at least one release image ref before upgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61412105253779644354560
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 3 · Sep 29: 1 · Sep 30: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000af62c0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-fs49jjvbxrlk" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9c1f57e-28e9-49ea-8226-45b731c2d8ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9c1f57e-28e9-49ea-8226-45b731c2d8ca",
      "name": "e9c1f57e-28e9-49ea-8226-45b731c2d8ca",
      "status": "Failed",
      "startTime": "2026-09-30T16:53:07.232391508Z",
      "endTime": "2026-09-30T17:22:15.941466136Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m42s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000af62c0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-fs49jjvbxrlk" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9c1f57e-28e9-49ea-8226-45b731c2d8ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9c1f57e-28e9-49ea-8226-45b731c2d8ca",
      "name": "e9c1f57e-28e9-49ea-8226-45b731c2d8ca",
      "status": "Failed",
      "startTime": "2026-09-30T16:53:07.232391508Z",
      "endTime": "2026-09-30T17:22:15.941466136Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m42s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc0010a8280>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-ztrwgb4tgx4r" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d",
      "name": "fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:39.354643704Z",
      "endTime": "2026-09-30T17:21:45.995494198Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m45s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc0010a8280>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-ztrwgb4tgx4r" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d",
      "name": "fb9dfc1a-3b4b-487f-b91c-0f6b1985bf0d",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:39.354643704Z",
      "endTime": "2026-09-30T17:21:45.995494198Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 25m45s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000f1a380>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-rwh7s9rvp89r" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44",
      "name": "eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:15.764760069Z",
      "endTime": "2026-09-30T17:21:16.086415783Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 24m59s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000f1a380>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-rwh7s9rvp89r" to finish creating: GET https://rp.j5446016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44",
      "name": "eb690e90-f4f6-4eb4-9d4f-fe046ccd3d44",
      "status": "Failed",
      "startTime": "2026-09-30T16:52:15.764760069Z",
      "endTime": "2026-09-30T17:21:16.086415783Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.35 is Partial (want Completed), started 24m59s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #71782105327789455446016
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2 · Sep 29: 0 · Sep 30: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000c681a0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-fl5kpw" in resourcegroup="rg-np-version-upgrade-fl5kpw-96mvvhr695cd" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0",
      "name": "a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0",
      "status": "Failed",
      "startTime": "2026-09-30T12:36:12.039132485Z",
      "endTime": "2026-09-30T12:55:21.071689427Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000c681a0>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-fl5kpw" in resourcegroup="rg-np-version-upgrade-fl5kpw-96mvvhr695cd" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0",
      "name": "a0ac539a-44de-4d6b-be03-d9b6fc3ccdc0",
      "status": "Failed",
      "startTime": "2026-09-30T12:36:12.039132485Z",
      "endTime": "2026-09-30T12:55:21.071689427Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000abb5a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-dr2mph" in resourcegroup="rg-np-noedge-dr2mph-j7ng55t8mcf6" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/78b330e2-3532-4f23-91f1-b24676955611
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/78b330e2-3532-4f23-91f1-b24676955611",
      "name": "78b330e2-3532-4f23-91f1-b24676955611",
      "status": "Failed",
      "startTime": "2026-09-30T12:38:52.851263491Z",
      "endTime": "2026-09-30T12:58:01.083392787Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000abb5a0>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-dr2mph" in resourcegroup="rg-np-noedge-dr2mph-j7ng55t8mcf6" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/78b330e2-3532-4f23-91f1-b24676955611
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/78b330e2-3532-4f23-91f1-b24676955611",
      "name": "78b330e2-3532-4f23-91f1-b24676955611",
      "status": "Failed",
      "startTime": "2026-09-30T12:38:52.851263491Z",
      "endTime": "2026-09-30T12:58:01.083392787Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-889bmn" for cluster "cluster-candidate-4-20-889bmn"
Unexpected error:
    <*fmt.wrapError | 0xc0015841e0>: 
    failed to create NodePool np-889bmn: failed waiting for nodepool="np-889bmn" for cluster "cluster-candidate-4-20-889bmn" in resourcegroup="rg-candidate-4-20-qcvff5h4wj94" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e30a66c5-dd1a-49f7-a04b-0cec3c58378a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e30a66c5-dd1a-49f7-a04b-0cec3c58378a",
      "name": "e30a66c5-dd1a-49f7-a04b-0cec3c58378a",
      "status": "Failed",
      "startTime": "2026-09-30T12:19:11.883646471Z",
      "endTime": "2026-09-30T12:38:11.728721311Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-889bmn" for cluster "cluster-candidate-4-20-889bmn"
Unexpected error:
    <*fmt.wrapError | 0xc0015841e0>: 
    failed to create NodePool np-889bmn: failed waiting for nodepool="np-889bmn" for cluster "cluster-candidate-4-20-889bmn" in resourcegroup="rg-candidate-4-20-qcvff5h4wj94" to finish creating: GET https://rp.j4354560.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e30a66c5-dd1a-49f7-a04b-0cec3c58378a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e30a66c5-dd1a-49f7-a04b-0cec3c58378a",
      "name": "e30a66c5-dd1a-49f7-a04b-0cec3c58378a",
      "status": "Failed",
      "startTime": "2026-09-30T12:19:11.883646471Z",
      "endTime": "2026-09-30T12:38:11.728721311Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: No Machines are created; node pool condition AllMachinesReady is False: No Machines are created"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61412105253779644354560
expected node pool nodes to report at least one release image ref before downgrade
full failure pattern:
expected node pool nodes to report at least one release image ref before downgrade
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 2 · Sep 29: 0 · Sep 30: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:615]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:615]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:755]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:755]: expected node pool nodes to report at least one release image ref before downgrade
Expected
    <set.Set[string] | len:0>: ...
not to be empty
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61412105253779644354560
alert [svc] ServiceMemoryDrift fired
full failure pattern:
alert [svc] ServiceMemoryDrift fired
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
alert11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsSep 24: 28 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-30T09:55:42Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j0283264-mgmt-2", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-84tpd", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-84tpd (namespace arobit) on cluster ci01-j0283264-mgmt-2 has averaged 123.4% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72612105187789510283264
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 41 · Sep 30: 1PROD, STG
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc00082e468>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc00082e468>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61412105253779644354560
ERROR CODE: DeploymentNotFound
full failure pattern:
ERROR CODE: DeploymentNotFound
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:130]: failed to create customer resources for aggregated feature cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009224e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-agg-cluster-kv5cgd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-agg-cluster-kv5cgd
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-agg-cluster-kv5cgd' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:130]: failed to create customer resources for aggregated feature cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009224e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-agg-cluster-kv5cgd" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-agg-cluster-kv5cgd
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: DeploymentNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentNotFound",
        "message": "Deployment 'mi-agg-cluster-kv5cgd' could not be found."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72292105107374309117952
ERROR CODE: DeploymentFailed; detail code TaskCanceledException; detail message Error while processing network security perimeter association: The request was canceled due to the ...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code TaskCanceledException; detail message Error while processing network security perimeter association: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.; provider Microsoft.Network
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
provision11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
time=2026-09-30T20:31:01.398Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-09-30T20:31:01.965Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2
time=2026-09-30T20:31:02.790Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 deployment=0a6bcb1a4c8db820be60ebb3431d76a20397d50a3cee281e89a5518a17e2b3e0 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j5173888-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F0a6bcb1a4c8db820be60ebb3431d76a20397d50a3cee281e89a5518a17e2b3e0
time=2026-09-30T20:33:03.396Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5173888-mgmt-2/providers/Microsoft.Resources/deployments/0a6bcb1a4c8db820be60ebb3431d76a20397d50a3cee281e89a5518a17e2b3e0/operationStatuses/08584108066230621918\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5173888-mgmt-2/providers/Microsoft.Resources/deployments/profile-mlyktaqazs6u6\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5173888-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j5173888-mgmt-2/resourceAssociations/nsp-j5173888-mgmt-2-ra-30be3175-d83b-506a-923e-30e4da745648\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"TaskCanceledException\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5173888-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j5173888-mgmt-2/resourceAssociations/nsp-j5173888-mgmt-2-ra-30be3175-d83b-506a-923e-30e4da745648, Error message: The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nsp (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69932105384924235173888
expected all nodes to have label 'key1=value1'
full failure pattern:
expected all nodes to have label 'key1=value1'
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:158]: expected all nodes to have label 'key1=value1'
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:158]: expected all nodes to have label 'key1=value1'
Expected
    <bool>: ...
to be true
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #61412105253779644354560
failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress context deadline exceed...
full failure pattern:
failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
provision11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
time=2026-09-30T10:09:47.479Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 description="Step storageclass\n Kind: Helm\n"
time=2026-09-30T10:09:48.123Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 values="map[skuName:PremiumV2_LRS unannotateJobImage:mcr.microsoft.com/aks/command/runtime@sha256:161971cdd3bfe66be6a9b96d67fabf8c4b54a88b841add1aaced26b0a00358b4]"
time=2026-09-30T10:09:48.123Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-09-30T10:09:48.426Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 namespace=default-sc
time=2026-09-30T10:09:48.816Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 dryRun=false
time=2026-09-30T10:09:48.816Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-09-30T10:09:48.942Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2
time=2026-09-30T10:14:53.371Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
time=2026-09-30T10:14:53.371Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=storageclass stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/default-sc/unannotate-default-sc not ready. status: InProgress, message: Job in progress\ncontext deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/storageclass (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #70712105227392430641152
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011481e0>: 
    resource group "rg-cluster-back-version-cnrv4422rs76": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-cnrv4422rs76", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-cnrv4422rs76" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-cnrv4422rs76, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-cnrv4422rs76" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-cnrv4422rs76, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0011481e0>: 
    resource group "rg-cluster-back-version-cnrv4422rs76": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-cnrv4422rs76", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-cnrv4422rs76" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-cnrv4422rs76, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-cnrv4422rs76" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-cnrv4422rs76, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72612105187789510283264
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000ea00e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-7mqggzhqq55s" to finish creating: GET https://rp.j0694272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ec169ee-a122-45a3-a622-b7083455fb90
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ec169ee-a122-45a3-a622-b7083455fb90",
      "name": "1ec169ee-a122-45a3-a622-b7083455fb90",
      "status": "Failed",
      "startTime": "2026-09-30T14:16:43.796681639Z",
      "endTime": "2026-09-30T14:35:52.497968724Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5o8a1j8f7l2m2y-np-1-qdcxj-hwbpg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5o8a1j8f7l2m2y-np-1-qdcxj-hwbpg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000ea00e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-7mqggzhqq55s" to finish creating: GET https://rp.j0694272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ec169ee-a122-45a3-a622-b7083455fb90
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ec169ee-a122-45a3-a622-b7083455fb90",
      "name": "1ec169ee-a122-45a3-a622-b7083455fb90",
      "status": "Failed",
      "startTime": "2026-09-30T14:16:43.796681639Z",
      "endTime": "2026-09-30T14:35:52.497968724Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y5o8a1j8f7l2m2y-np-1-qdcxj-hwbpg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y5o8a1j8f7l2m2y-np-1-qdcxj-hwbpg: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #70532105279807200694272
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0014f6680>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-mxcczjbklf6k" to finish creating: GET https://rp.j1701248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49794eca-872f-4949-8230-a9c6d512647a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49794eca-872f-4949-8230-a9c6d512647a",
      "name": "49794eca-872f-4949-8230-a9c6d512647a",
      "status": "Failed",
      "startTime": "2026-09-30T19:06:11.188459377Z",
      "endTime": "2026-09-30T19:25:16.297661458Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine p5d6o0j0i5s4u7a-autoscale-noaz-686hx-dvr6p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine p5d6o0j0i5s4u7a-autoscale-noaz-686hx-dvr6p: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0014f6680>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-mxcczjbklf6k" to finish creating: GET https://rp.j1701248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49794eca-872f-4949-8230-a9c6d512647a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/49794eca-872f-4949-8230-a9c6d512647a",
      "name": "49794eca-872f-4949-8230-a9c6d512647a",
      "status": "Failed",
      "startTime": "2026-09-30T19:06:11.188459377Z",
      "endTime": "2026-09-30T19:25:16.297661458Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine p5d6o0j0i5s4u7a-autoscale-noaz-686hx-dvr6p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine p5d6o0j0i5s4u7a-autoscale-noaz-686hx-dvr6p: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72612105354557851701248
UpdateHCPCluster20260901 timed out after <minutes> minutes; context deadline exceeded
full failure pattern:
UpdateHCPCluster20260901 timed out after <minutes> minutes; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e11.02%1 of 98 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:301]: failed to update cluster with new KMS key with the second rotation
Unexpected error:
    <*fmt.wrapErrors | 0xc000dfb3e0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-vxcvln6bjmck" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-vxcvln6bjmck, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:301]: failed to update cluster with new KMS key with the second rotation
Unexpected error:
    <*fmt.wrapErrors | 0xc000dfb3e0>: 
    failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-vxcvln6bjmck" to finish updating, caused by: timeout '23.000000' minutes exceeded during UpdateHCPCluster20260901 for cluster kms-key-rotate-422 in resource group kms-key-rotate-vxcvln6bjmck, error: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72612105187789510283264

Environment: INT

Failure patterns: 3 · Failures matched: 13 · Total runs: 8 · Runs affected: 3

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e225.00%2 of 8 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 1 · Sep 28: 1 · Sep 29: 4 · Sep 30: 2DEV, PROD, STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d6f340>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-t8kp5gffh7ph" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e",
      "name": "fbb7c321-b5c6-4145-9be8-48a64282647e",
      "status": "Failed",
      "startTime": "2026-09-30T22:47:02.188860165Z",
      "endTime": "2026-09-30T23:06:03.834732494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d6f340>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-t8kp5gffh7ph" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fbb7c321-b5c6-4145-9be8-48a64282647e",
      "name": "fbb7c321-b5c6-4145-9be8-48a64282647e",
      "status": "Failed",
      "startTime": "2026-09-30T22:47:02.188860165Z",
      "endTime": "2026-09-30T23:06:03.834732494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine g1n9i9o3o7q1q9w-np-1-42h49-pdfcq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-t8kp5gffh7ph--managed/providers/Microsoft.Compute/virtualMachines/g1n9i9o3o7q1q9w-np-1-42h49-pdfcq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc001088480>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-mmgvqz" in resourcegroup="rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846",
      "name": "b1caf488-0088-4fa0-8a37-c1f30126d846",
      "status": "Failed",
      "startTime": "2026-09-30T20:37:05.655658972Z",
      "endTime": "2026-09-30T20:56:13.566012243Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc001088480>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-mmgvqz" in resourcegroup="rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b1caf488-0088-4fa0-8a37-c1f30126d846",
      "name": "b1caf488-0088-4fa0-8a37-c1f30126d846",
      "status": "Failed",
      "startTime": "2026-09-30T20:37:05.655658972Z",
      "endTime": "2026-09-30T20:56:13.566012243Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mmgvqz-rdcrv8s7btgd-np-upgrade-mm-8330a333/providers/Microsoft.Compute/virtualMachines/u6u2x7j9c3m3v3g-npupgrade-4-20-vxvrm-whh28 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2105422878328492032
n/a2105375087707295744
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value canno...
full failure pattern:
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e112.50%1 of 8 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 10none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-b9ld4k" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000a93bf0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-m7g7x4rhlkmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-b9ld4k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-b9ld4k" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000a93bf0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-m7g7x4rhlkmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-b9ld4k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-52n697" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0015481e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-52jx5pj75l4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-52n697
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-52n697" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc0015481e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-52jx5pj75l4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-52n697
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-nkn9cp" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000fb60c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-2s89ppfpwxxt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-nkn9cp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-nkn9cp" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000fb60c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-2s89ppfpwxxt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-nkn9cp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (10)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105422878328492032
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible tha...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url>
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e112.50%1 of 8 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:112]: failed to create customer resources for cluster "cluster-candidate-4-20-t74pss"
Unexpected error:
    <*fmt.wrapError | 0xc000b40260>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-20-t74pss-7wvjkj" in resourcegroup="rg-candidate-4-20-bmfs7fmklpfb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-bmfs7fmklpfb/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-t74pss-7wvjkj/operationStatuses/08584108180410013833
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-cr5ick5opsuzu' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:112]: failed to create customer resources for cluster "cluster-candidate-4-20-t74pss"
Unexpected error:
    <*fmt.wrapError | 0xc000b40260>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-20-t74pss-7wvjkj" in resourcegroup="rg-candidate-4-20-bmfs7fmklpfb" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-20-bmfs7fmklpfb/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-20-t74pss-7wvjkj/operationStatuses/08584108180410013833
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"VaultAlreadyExists\",\r\n    \"message\": \"The vault name 'cust-kv-cr5ick5opsuzu' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105336380325367808

Environment: PROD

Failure patterns: 4 · Failures matched: 10 · Total runs: 3 · Runs affected: 2

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e266.67%2 of 3 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 24: 1 · Sep 25: 1 · Sep 26: 2 · Sep 27: 2 · Sep 28: 1 · Sep 29: 3 · Sep 30: 2DEV, STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:178]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc001069ba8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-29-171841, targetMinor=5.1.0-0.nightly-multi-2026-09-30-105558) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-30-105558"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:178]: Timed out after 2700.007s.
Expected success, but got an error:
    <*errors.joinError | 0xc001069ba8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-29-171841, targetMinor=5.1.0-0.nightly-multi-2026-09-30-105558) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-30-105558"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc001415050>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc001415050>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2105432621860261888
n/a2105115529034338304
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e133.33%1 of 3 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 24: 4 · Sep 25: 4 · Sep 26: 2 · Sep 27: 4 · Sep 28: 4 · Sep 29: 4 · Sep 30: 4STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc000d541a0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-mv9rpr" in resourcegroup="rg-np-skip-minor-mv9rpr-lccf5r2wcvcf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-mv9rpr-lccf5r2wcvcf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-mv9rpr/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc000d541a0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-mv9rpr" in resourcegroup="rg-np-skip-minor-mv9rpr-lccf5r2wcvcf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-mv9rpr-lccf5r2wcvcf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-mv9rpr/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fzw8hf" for cluster "cluster-nightly-4-20-fzw8hf"
Unexpected error:
    <*fmt.wrapError | 0xc000ca4c80>: 
    failed to create NodePool np-fzw8hf: failed starting nodepool creation "np-fzw8hf" for cluster "cluster-nightly-4-20-fzw8hf" in resourcegroup="rg-nightly-4-20-tfqsvtwn4ns7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-tfqsvtwn4ns7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-fzw8hf/nodePools/np-fzw8hf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-014938\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fzw8hf" for cluster "cluster-nightly-4-20-fzw8hf"
Unexpected error:
    <*fmt.wrapError | 0xc000ca4c80>: 
    failed to create NodePool np-fzw8hf: failed starting nodepool creation "np-fzw8hf" for cluster "cluster-nightly-4-20-fzw8hf" in resourcegroup="rg-nightly-4-20-tfqsvtwn4ns7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-tfqsvtwn4ns7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-fzw8hf/nodePools/np-fzw8hf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-014938\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc001472300>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4zkcv5" resourcegroup="rg-np-dg-minor-4zkcv5-pjsq2w8jb2dl": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4zkcv5-pjsq2w8jb2dl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4zkcv5/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc001472300>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4zkcv5" resourcegroup="rg-np-dg-minor-4zkcv5-pjsq2w8jb2dl": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4zkcv5-pjsq2w8jb2dl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4zkcv5/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105432621860261888
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e266.67%2 of 3 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 4 · Sep 29: 10 · Sep 30: 2DEV, INT, STG
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0013120e0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-sttxg5n27hh5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/95422086-7eb5-4fab-930f-f78b5c9185bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/95422086-7eb5-4fab-930f-f78b5c9185bc",
      "name": "95422086-7eb5-4fab-930f-f78b5c9185bc",
      "status": "Failed",
      "startTime": "2026-10-01T00:05:50.759671414Z",
      "endTime": "2026-10-01T00:24:59.018405341Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sttxg5n27hh5--managed/providers/Microsoft.Compute/virtualMachines/m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sttxg5n27hh5--managed/providers/Microsoft.Compute/virtualMachines/m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0013120e0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-sttxg5n27hh5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/95422086-7eb5-4fab-930f-f78b5c9185bc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/95422086-7eb5-4fab-930f-f78b5c9185bc",
      "name": "95422086-7eb5-4fab-930f-f78b5c9185bc",
      "status": "Failed",
      "startTime": "2026-10-01T00:05:50.759671414Z",
      "endTime": "2026-10-01T00:24:59.018405341Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sttxg5n27hh5--managed/providers/Microsoft.Compute/virtualMachines/m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-sttxg5n27hh5--managed/providers/Microsoft.Compute/virtualMachines/m8y8w5f8k4s4f3b-autoscale-noaz-xcklm-vq2zl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc0009c85c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-wbpxbvspwkdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/44b3f004-e16f-436f-bdc7-4652956b17b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/44b3f004-e16f-436f-bdc7-4652956b17b0",
      "name": "44b3f004-e16f-436f-bdc7-4652956b17b0",
      "status": "Failed",
      "startTime": "2026-09-30T02:12:50.954793928Z",
      "endTime": "2026-09-30T02:31:50.125456331Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-mhjdk-jdk87: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-wbpxbvspwkdl--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mhjdk-jdk87 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-mhjdk-jdk87: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-wbpxbvspwkdl--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mhjdk-jdk87 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc0009c85c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-wbpxbvspwkdl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/44b3f004-e16f-436f-bdc7-4652956b17b0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/44b3f004-e16f-436f-bdc7-4652956b17b0",
      "name": "44b3f004-e16f-436f-bdc7-4652956b17b0",
      "status": "Failed",
      "startTime": "2026-09-30T02:12:50.954793928Z",
      "endTime": "2026-09-30T02:31:50.125456331Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-mhjdk-jdk87: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-wbpxbvspwkdl--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mhjdk-jdk87 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20260901-np-1-mhjdk-jdk87: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-wbpxbvspwkdl--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mhjdk-jdk87 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2105432621860261888
n/a2105115529034338304
VerifyBasicAccess failed: failed to list services: Unauthorized
full failure pattern:
VerifyBasicAccess failed: failed to list services: Unauthorized
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e133.33%1 of 3 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 1 · Sep 30: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:198]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
    <*errors.joinError | 0xc001619020>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:198]: failed to verify HCP cluster viability for patch-tags test
Unexpected error:
    <*errors.joinError | 0xc001619020>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:148]: failed to verify cluster "cp-ystream-upgrade-5-0-b96xbr" is viable before upgrade
Unexpected error:
    <*errors.joinError | 0xc000e76f48>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:148]: failed to verify cluster "cp-ystream-upgrade-5-0-b96xbr" is viable before upgrade
Unexpected error:
    <*errors.joinError | 0xc000e76f48>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105432621860261888

Environment: STG

Failure patterns: 7 · Failures matched: 21 · Total runs: 5 · Runs affected: 5

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e120.00%1 of 5 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 24: 3 · Sep 25: 4 · Sep 26: 2 · Sep 27: 1 · Sep 28: 4 · Sep 29: 4 · Sep 30: 4PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc00154c140>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-49bq5x" resourcegroup="rg-np-dg-minor-49bq5x-7f6vt9z24c7x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-49bq5x-7f6vt9z24c7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-49bq5x/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc00154c140>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-49bq5x" resourcegroup="rg-np-dg-minor-49bq5x-7f6vt9z24c7x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-49bq5x-7f6vt9z24c7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-49bq5x/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-prg87r" for cluster "cluster-nightly-4-20-prg87r"
Unexpected error:
    <*fmt.wrapError | 0xc0014907c0>: 
    failed to create NodePool np-prg87r: failed starting nodepool creation "np-prg87r" for cluster "cluster-nightly-4-20-prg87r" in resourcegroup="rg-nightly-4-20-kjrkpmc2vcnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-kjrkpmc2vcnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-prg87r/nodePools/np-prg87r
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-014938\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-prg87r" for cluster "cluster-nightly-4-20-prg87r"
Unexpected error:
    <*fmt.wrapError | 0xc0014907c0>: 
    failed to create NodePool np-prg87r: failed starting nodepool creation "np-prg87r" for cluster "cluster-nightly-4-20-prg87r" in resourcegroup="rg-nightly-4-20-kjrkpmc2vcnb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-kjrkpmc2vcnb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-prg87r/nodePools/np-prg87r
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-014938\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc000b92100>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-qglcmw" in resourcegroup="rg-np-version-upgrade-qglcmw-2xxcc62c56zb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-qglcmw-2xxcc62c56zb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-qglcmw/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-30-185324
Unexpected error:
    <*fmt.wrapError | 0xc000b92100>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-qglcmw" in resourcegroup="rg-np-version-upgrade-qglcmw-2xxcc62c56zb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-qglcmw-2xxcc62c56zb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-qglcmw/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-30-185324\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105432621902204928
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
full failure pattern:
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e120.00%1 of 5 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsSep 24: 1 · Sep 25: 1 · Sep 26: 2 · Sep 27: 2 · Sep 28: 1 · Sep 29: 1 · Sep 30: 1DEV, PROD
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016d54b8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.003s.
Expected success, but got an error:
    <*errors.joinError | 0xc0016d54b8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0, targetMinor=5.1) failed: clusterversion status.history has no version in target minor "5.1"
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105115529080475648
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
e2e360.00%3 of 5 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 17 · Sep 27: 1 · Sep 28: 2 · Sep 29: 1 · Sep 30: 10DEV, INT, PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc0012c2500>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-xhfvnf" in resourcegroup="rg-np-noedge-xhfvnf-d6cdpdqmxzz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
      "name": "79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
      "status": "Failed",
      "startTime": "2026-09-30T17:20:42.49451817Z",
      "endTime": "2026-09-30T17:39:43.993225307Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc0012c2500>: 
    failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-xhfvnf" in resourcegroup="rg-np-noedge-xhfvnf-d6cdpdqmxzz5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
      "name": "79ccdc55-d7b6-4981-9ed0-35fdb60bc186",
      "status": "Failed",
      "startTime": "2026-09-30T17:20:42.49451817Z",
      "endTime": "2026-09-30T17:39:43.993225307Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-xhfvnf-d6cdpdqmxzz5-np-ne-xhfvnf--managed/providers/Microsoft.Compute/virtualMachines/h9k2r7c7n2q4l9a-npnoedge-4-20-pmpd9-ntn64 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-djrg27" for cluster "cluster-candidate-4-20-djrg27"
Unexpected error:
    <*fmt.wrapError | 0xc000119c40>: 
    failed to create NodePool np-djrg27: failed waiting for nodepool="np-djrg27" for cluster "cluster-candidate-4-20-djrg27" in resourcegroup="rg-candidate-4-20-hwlm5v4jsr9p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44",
      "name": "94e00bc9-969c-49a7-a690-ac544d6d6b44",
      "status": "Failed",
      "startTime": "2026-09-30T16:46:47.162303776Z",
      "endTime": "2026-09-30T17:05:53.284012473Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-djrg27" for cluster "cluster-candidate-4-20-djrg27"
Unexpected error:
    <*fmt.wrapError | 0xc000119c40>: 
    failed to create NodePool np-djrg27: failed waiting for nodepool="np-djrg27" for cluster "cluster-candidate-4-20-djrg27" in resourcegroup="rg-candidate-4-20-hwlm5v4jsr9p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/94e00bc9-969c-49a7-a690-ac544d6d6b44",
      "name": "94e00bc9-969c-49a7-a690-ac544d6d6b44",
      "status": "Failed",
      "startTime": "2026-09-30T16:46:47.162303776Z",
      "endTime": "2026-09-30T17:05:53.284012473Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-hwlm5v4jsr9p-candidate-djrg27--managed/providers/Microsoft.Compute/virtualMachines/d7d8e3i5j6c6k1p-np-djrg27-4sr5k-bwq7t to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc0015c0ae0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-twtdf7gdskgb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512",
      "name": "ffbf36ab-0bd9-4c0c-8698-3035c7937512",
      "status": "Failed",
      "startTime": "2026-09-30T16:25:28.987424124Z",
      "endTime": "2026-09-30T16:44:30.182558055Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-njlb6-5n7mv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-njlb6-5n7mv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc0015c0ae0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-twtdf7gdskgb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ffbf36ab-0bd9-4c0c-8698-3035c7937512",
      "name": "ffbf36ab-0bd9-4c0c-8698-3035c7937512",
      "status": "Failed",
      "startTime": "2026-09-30T16:25:28.987424124Z",
      "endTime": "2026-09-30T16:44:30.182558055Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-njlb6-5n7mv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-njlb6-5n7mv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-twtdf7gdskgb--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-njlb6-5n7mv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (9)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest2
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to create an HCP cluster then delete it by deleting the customer resource group1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallelCustomer should create an HCP cluster and validate TLS certificates1
Affected runs (3)
Date (UTC)Associated PRProw job
n/a2105324689185312768
n/a2105269033774354432
n/a2105115529080475648
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e240.00%2 of 5 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 1 · Sep 29: 0 · Sep 30: 2DEV
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00148e9c0>: 
    resource group "rg-zstream-upgrade-5-0-l6m7gzggxh5z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-mrptx7" in resourcegroup="rg-zstream-upgrade-5-0-l6m7gzggxh5z" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5",
      "name": "d96cdefb-0420-40da-ba9f-dde8391c74b5",
      "status": "Failed",
      "startTime": "2026-09-30T13:28:47.89996524Z",
      "endTime": "2026-09-30T13:52:47.080528249Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5bj2pp5rm42moqm553gsgpuq5afnl4 still exists (deletion dispatched at 2026-09-30T13:28:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00148e9c0>: 
    resource group "rg-zstream-upgrade-5-0-l6m7gzggxh5z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-mrptx7" in resourcegroup="rg-zstream-upgrade-5-0-l6m7gzggxh5z" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d96cdefb-0420-40da-ba9f-dde8391c74b5",
      "name": "d96cdefb-0420-40da-ba9f-dde8391c74b5",
      "status": "Failed",
      "startTime": "2026-09-30T13:28:47.89996524Z",
      "endTime": "2026-09-30T13:52:47.080528249Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t5bj2pp5rm42moqm553gsgpuq5afnl4 still exists (deletion dispatched at 2026-09-30T13:28:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000a8a4b0>: 
    resource group "rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-b6jrz8" in resourcegroup="rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
      "name": "8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
      "status": "Failed",
      "startTime": "2026-09-30T08:07:19.758629115Z",
      "endTime": "2026-09-30T08:31:25.87901072Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56tcneq332c0ht4fv2g22jot5kdluc still exists (deletion dispatched at 2026-09-30T08:07:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000a8a4b0>: 
    resource group "rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-b6jrz8" in resourcegroup="rg-cp-ystream-upgrade-5-0-sxzh6sqdlxpm" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
      "name": "8dac7cbd-9195-4d5c-adc5-24fab76d3fc6",
      "status": "Failed",
      "startTime": "2026-09-30T08:07:19.758629115Z",
      "endTime": "2026-09-30T08:31:25.87901072Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t56tcneq332c0ht4fv2g22jot5kdluc still exists (deletion dispatched at 2026-09-30T08:07:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2105269033774354432
n/a2105187694681264128
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e240.00%2 of 5 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 2DEV
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001116780>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-vh7hggp8cw8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba",
      "name": "eed44f57-1a91-4587-8901-935588d355ba",
      "status": "Failed",
      "startTime": "2026-09-30T17:09:44.769920829Z",
      "endTime": "2026-09-30T17:28:44.009160044Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001116780>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-vh7hggp8cw8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eed44f57-1a91-4587-8901-935588d355ba",
      "name": "eed44f57-1a91-4587-8901-935588d355ba",
      "status": "Failed",
      "startTime": "2026-09-30T17:09:44.769920829Z",
      "endTime": "2026-09-30T17:28:44.009160044Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-vh7hggp8cw8p--managed/providers/Microsoft.Compute/virtualMachines/l0v3x1l5h8o3n9c-np-deletable-cmqdm-h6zzh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0013950a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-6stx4mwz4j4k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35",
      "name": "626e713d-65ca-4af7-8b7c-fccfc5b04f35",
      "status": "Failed",
      "startTime": "2026-09-30T08:05:30.277326796Z",
      "endTime": "2026-09-30T08:24:30.136180053Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0013950a0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-6stx4mwz4j4k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/626e713d-65ca-4af7-8b7c-fccfc5b04f35",
      "name": "626e713d-65ca-4af7-8b7c-fccfc5b04f35",
      "status": "Failed",
      "startTime": "2026-09-30T08:05:30.277326796Z",
      "endTime": "2026-09-30T08:24:30.136180053Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-6stx4mwz4j4k--managed/providers/Microsoft.Compute/virtualMachines/u9g6k2x0j4x3g3x-np-deletable-k7zvr-ljwvr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to delete a nodepool2
Affected runs (2)
Date (UTC)Associated PRProw job
n/a2105324689185312768
n/a2105187694681264128
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [openshift-apiserver]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e120.00%1 of 5 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-srbb2hbvsczt/cluster-nightly-5-1-vxb9dm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c1af40>: 
    failed to create HCP cluster cluster-nightly-5-1-vxb9dm: failed waiting for cluster="cluster-nightly-5-1-vxb9dm" in resourcegroup="rg-nightly-5-1-srbb2hbvsczt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
      "name": "fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
      "status": "Failed",
      "startTime": "2026-09-30T23:26:50.189510673Z",
      "endTime": "2026-09-30T23:55:57.188151115Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-0.nightly-multi-2026-09-30-105558 is Partial (want Completed), started 23m56s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-srbb2hbvsczt/cluster-nightly-5-1-vxb9dm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c1af40>: 
    failed to create HCP cluster cluster-nightly-5-1-vxb9dm: failed waiting for cluster="cluster-nightly-5-1-vxb9dm" in resourcegroup="rg-nightly-5-1-srbb2hbvsczt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
      "name": "fc96c037-ee80-4be7-a7e9-fa4828e28e0b",
      "status": "Failed",
      "startTime": "2026-09-30T23:26:50.189510673Z",
      "endTime": "2026-09-30T23:55:57.188151115Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.1.0-0.nightly-multi-2026-09-30-105558 is Partial (want Completed), started 23m56s ago; hosted cluster degraded: UnavailableReplicas: openshift-apiserver deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105432621902204928
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
full failure pattern:
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e120.00%1 of 5 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsSep 24: 0 · Sep 25: 0 · Sep 26: 0 · Sep 27: 0 · Sep 28: 0 · Sep 29: 0 · Sep 30: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cd79a0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-ldfdqgv9q9rv" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cd79a0>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-ldfdqgv9q9rv" to finish creating: context deadline exceeded
    ...
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2105269033774354432