No failure patterns match your search.
Environment: STG
Failure patterns: 4 · Failures matched: 36 · Total runs: 3 · Runs affected: 3
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 3 | 100.00%3 of 3 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsJun 26: 0 · Jun 27: 0 · Jun 28: 0 · Jun 29: 0 · Jun 30: 0 · Jul 1: 0 · Jul 2: 7 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapErrors | 0xc0026f1410>:
failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-45ldvl, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-45ldvl" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapErrors | 0xc0026f1410>:
failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-45ldvl, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-45ldvl" to finish creating: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:148]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapErrors | 0xc002a0b620>:
failed to create NodePool autoscale-az, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool autoscale-az in resource group np-autoscaling-n9df2b, error: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-n9df2b" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:148]: failed to create AZ nodepool autoscale-az with autoscaling
Unexpected error:
<*fmt.wrapErrors | 0xc002a0b620>:
failed to create NodePool autoscale-az, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool autoscale-az in resource group np-autoscaling-n9df2b, error: failed waiting for nodepool="autoscale-az" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-n9df2b" to finish creating: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-8skd5b" for cluster "cluster-nightly-4-20-8skd5b"
Unexpected error:
<*fmt.wrapErrors | 0xc0028a7680>:
failed to create NodePool np-8skd5b, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-8skd5b in resource group rg-nightly-4-20-dvmp6t, error: failed waiting for nodepool="np-8skd5b" for cluster "cluster-nightly-4-20-8skd5b" in resourcegroup="rg-nightly-4-20-dvmp6t" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-8skd5b" for cluster "cluster-nightly-4-20-8skd5b"
Unexpected error:
<*fmt.wrapErrors | 0xc0028a7680>:
failed to create NodePool np-8skd5b, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-8skd5b in resource group rg-nightly-4-20-dvmp6t, error: failed waiting for nodepool="np-8skd5b" for cluster "cluster-nightly-4-20-8skd5b" in resourcegroup="rg-nightly-4-20-dvmp6t" to finish creating: context deadline exceeded
...
occurredContributing tests (7)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 2 | 66.67%2 of 3 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsJun 26: 0 · Jun 27: 0 · Jun 28: 0 · Jun 29: 0 · Jun 30: 0 · Jul 1: 0 · Jul 2: 23 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-4-23-mw7g87/cluster-nightly-4-23-b45mnx should provision
Unexpected error:
<*fmt.wrapErrors | 0xc000a30de0>:
failed to create HCP cluster cluster-nightly-4-23-b45mnx, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-b45mnx in resource group rg-nightly-4-23-mw7g87, error: failed waiting for cluster="cluster-nightly-4-23-b45mnx" in resourcegroup="rg-nightly-4-23-mw7g87" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-b45mnx in resource group rg-nightly-4-23-mw7g87, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-4-23-mw7g87/cluster-nightly-4-23-b45mnx should provision
Unexpected error:
<*fmt.wrapErrors | 0xc000a30de0>:
failed to create HCP cluster cluster-nightly-4-23-b45mnx, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-b45mnx in resource group rg-nightly-4-23-mw7g87, error: failed waiting for cluster="cluster-nightly-4-23-b45mnx" in resourcegroup="rg-nightly-4-23-mw7g87" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-b45mnx in resource group rg-nightly-4-23-mw7g87, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-5-0-mn6b27/cluster-nightly-5-0-7sf6qf should provision
Unexpected error:
<*fmt.wrapErrors | 0xc001486090>:
failed to create HCP cluster cluster-nightly-5-0-7sf6qf, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-7sf6qf in resource group rg-nightly-5-0-mn6b27, error: failed waiting for cluster="cluster-nightly-5-0-7sf6qf" in resourcegroup="rg-nightly-5-0-mn6b27" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-7sf6qf in resource group rg-nightly-5-0-mn6b27, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-5-0-mn6b27/cluster-nightly-5-0-7sf6qf should provision
Unexpected error:
<*fmt.wrapErrors | 0xc001486090>:
failed to create HCP cluster cluster-nightly-5-0-7sf6qf, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-7sf6qf in resource group rg-nightly-5-0-mn6b27, error: failed waiting for cluster="cluster-nightly-5-0-7sf6qf" in resourcegroup="rg-nightly-5-0-mn6b27" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-5-0-7sf6qf in resource group rg-nightly-5-0-mn6b27, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-srwxkf/cluster-candidate-5-0-x94n2m should provision
Unexpected error:
<*fmt.wrapErrors | 0xc000f8ec60>:
failed to create HCP cluster cluster-candidate-5-0-x94n2m, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-x94n2m in resource group rg-candidate-5-0-srwxkf, error: failed waiting for cluster="cluster-candidate-5-0-x94n2m" in resourcegroup="rg-candidate-5-0-srwxkf" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-x94n2m in resource group rg-candidate-5-0-srwxkf, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-5-0-srwxkf/cluster-candidate-5-0-x94n2m should provision
Unexpected error:
<*fmt.wrapErrors | 0xc000f8ec60>:
failed to create HCP cluster cluster-candidate-5-0-x94n2m, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-x94n2m in resource group rg-candidate-5-0-srwxkf, error: failed waiting for cluster="cluster-candidate-5-0-x94n2m" in resourcegroup="rg-candidate-5-0-srwxkf" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-5-0-x94n2m in resource group rg-candidate-5-0-srwxkf, error: context deadline exceeded
...
occurredContributing tests (23)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 1 | 33.33%1 of 3 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsJun 26: 0 · Jun 27: 0 · Jun 28: 0 · Jun 29: 0 · Jun 30: 0 · Jul 1: 0 · Jul 2: 5 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00190c168>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-19-5wmrnm" in resourcegroup="rg-zstream-upgrade-4-19-l9cvg8" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00190c168>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-19-5wmrnm" in resourcegroup="rg-zstream-upgrade-4-19-l9cvg8" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000ccc2e8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-sn4mb4" in resourcegroup="rg-zstream-upgrade-4-20-4p2hrd" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000ccc2e8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-sn4mb4" in resourcegroup="rg-zstream-upgrade-4-20-4p2hrd" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00158e198>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m5x4n6" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00158e198>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-m5x4n6" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurredContributing tests (5)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
client rate limiter Wait returned an error: context deadline exceededfull failure pattern: client rate limiter Wait returned an error: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 33.33%1 of 3 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJun 26: 0 · Jun 27: 0 · Jun 28: 0 · Jun 29: 0 · Jun 30: 0 · Jul 1: 0 · Jul 2: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc001e38480>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-zvscj7, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-zvscj7" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc001e38480>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-zvscj7, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-zvscj7" to finish creating: context deadline exceeded
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||