No failure patterns match your search.
Environment: STG
Failure patterns: 5 · Failures matched: 37 · Total runs: 6 · Runs affected: 5
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 4 | 66.67%4 of 6 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 22 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-s2sv5c" with version 4.22.0 on candidate channel
Unexpected error:
<*fmt.wrapErrors | 0xc001520390>:
failed to create HCP cluster cluster-zstream-4-22-s2sv5c, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-s2sv5c in resource group rg-zstream-upgrade-4-22-9pz7hs, error: failed waiting for cluster="cluster-zstream-4-22-s2sv5c" in resourcegroup="rg-zstream-upgrade-4-22-9pz7hs" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-s2sv5c in resource group rg-zstream-upgrade-4-22-9pz7hs, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-s2sv5c" with version 4.22.0 on candidate channel
Unexpected error:
<*fmt.wrapErrors | 0xc001520390>:
failed to create HCP cluster cluster-zstream-4-22-s2sv5c, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-s2sv5c in resource group rg-zstream-upgrade-4-22-9pz7hs, error: failed waiting for cluster="cluster-zstream-4-22-s2sv5c" in resourcegroup="rg-zstream-upgrade-4-22-9pz7hs" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-s2sv5c in resource group rg-zstream-upgrade-4-22-9pz7hs, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:282]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapErrors | 0xc000f60630>:
failed to create HCP cluster node-limit-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-kj8xs5, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-kj8xs5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-kj8xs5, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:282]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
<*fmt.wrapErrors | 0xc000f60630>:
failed to create HCP cluster node-limit-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-kj8xs5, error: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-kj8xs5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster node-limit-cluster in resource group node-limit-test-kj8xs5, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:596]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapErrors | 0xc0012d4510>:
failed to create HCP cluster np-downgrade-cpvw84, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-downgrade-cpvw84 in resource group rg-np-downgrade-cpvw84-sv54gs, error: failed waiting for cluster="np-downgrade-cpvw84" in resourcegroup="rg-np-downgrade-cpvw84-sv54gs" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-downgrade-cpvw84 in resource group rg-np-downgrade-cpvw84-sv54gs, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:596]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapErrors | 0xc0012d4510>:
failed to create HCP cluster np-downgrade-cpvw84, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-downgrade-cpvw84 in resource group rg-np-downgrade-cpvw84-sv54gs, error: failed waiting for cluster="np-downgrade-cpvw84" in resourcegroup="rg-np-downgrade-cpvw84-sv54gs" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-downgrade-cpvw84 in resource group rg-np-downgrade-cpvw84-sv54gs, error: context deadline exceeded
...
occurredContributing tests (22)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 4 | 66.67%4 of 6 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 5 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc00210a510>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-kdtw5w, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-kdtw5w" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc00210a510>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-kdtw5w, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-kdtw5w" to finish creating: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-r7rvfr" for cluster "cluster-candidate-4-22-r7rvfr"
Unexpected error:
<*fmt.wrapErrors | 0xc0007a2c30>:
failed to create NodePool np-r7rvfr, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-r7rvfr in resource group rg-candidate-4-22-gtwnxg, error: failed waiting for nodepool="np-r7rvfr" for cluster "cluster-candidate-4-22-r7rvfr" in resourcegroup="rg-candidate-4-22-gtwnxg" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-r7rvfr" for cluster "cluster-candidate-4-22-r7rvfr"
Unexpected error:
<*fmt.wrapErrors | 0xc0007a2c30>:
failed to create NodePool np-r7rvfr, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-r7rvfr in resource group rg-candidate-4-22-gtwnxg, error: failed waiting for nodepool="np-r7rvfr" for cluster "cluster-candidate-4-22-r7rvfr" in resourcegroup="rg-candidate-4-22-gtwnxg" to finish creating: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapErrors | 0xc001555890>:
failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-xz2vs2, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-xz2vs2" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapErrors | 0xc001555890>:
failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-xz2vs2, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-xz2vs2" to finish creating: context deadline exceeded
...
occurredContributing tests (4)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | e2e | 2 | 33.33%2 of 6 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s) | Show trend detailsJun 13: 0 · Jun 14: 2 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 8 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00017acd8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-846tff" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00017acd8>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-846tff" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0013ea180>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6hj75l" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0013ea180>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6hj75l" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000cf47e0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rsq4mt" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc000cf47e0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rsq4mt" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurredContributing tests (8)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4full failure pattern: VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4 Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 16.67%1 of 6 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:124]: Timed out after 2700.003s.
Expected success, but got an error:
<*errors.joinError | 0xc000fadb48>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.19.33) failed: no version in clusterversion/version status.history is greater than initial "4.19.33"
...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:124]: Timed out after 2700.003s.
Expected success, but got an error:
<*errors.joinError | 0xc000fadb48>:
VerifyHostedControlPlaneZStreamUpgradeOnly(initial=4.19.33) failed: no version in clusterversion/version status.history is greater than initial "4.19.33"
...Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
couldn't get current server API group list: the server has asked for the client to provide credentialsfull failure pattern: couldn't get current server API group list: the server has asked for the client to provide credentials Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 16.67%1 of 6 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJun 13: 0 · Jun 14: 0 · Jun 15: 0 · Jun 16: 0 · Jun 17: 0 · Jun 18: 0 · Jun 19: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:380]: Timed out after 300.006s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:365 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
<*errors.errorString | 0xc000a5a270>:
E0619 10:12:30.439124 2434 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0619 10:12:30.456274 2434 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0619 10:12:30.472320 2434 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0619 10:12:30.488740 2434 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0619 10:12:30.509879 2434 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:380]: Timed out after 300.006s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:365 with:
kubectl should execute successfully from authorized VM with OIDC token
Unexpected error:
<*errors.errorString | 0xc000a5a270>:
E0619 10:12:30.439124 2434 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0619 10:12:30.456274 2434 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0619 10:12:30.472320 2434 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0619 10:12:30.488740 2434 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E0619 10:12:30.509879 2434 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
error: You must be logged in to the server (the server has asked for the client to provide credentials)
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||