No failure patterns match your search.
Environment: STG
Failure patterns: 7 · Failures matched: 71 · Total runs: 16 · Runs affected: 9
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | e2e | 4 | 25.00%4 of 16 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days | Show trend detailsMay 10: 0 · May 11: 1 · May 12: 32 · May 13: 28 · May 14: 0 · May 15: 0 · May 16: 0 · May 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapErrors | 0xc000850420>:
failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-pnfkm2, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-pnfkm2" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-pnfkm2, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapErrors | 0xc000850420>:
failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-pnfkm2, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-pnfkm2" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-pnfkm2, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
<*fmt.wrapErrors | 0xc0010545a0>:
failed to create HCP cluster cluster-zstream-4-21-flm2tp, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-flm2tp in resource group rg-zstream-upgrade-4-21-nh6gmd, error: failed waiting for cluster="cluster-zstream-4-21-flm2tp" in resourcegroup="rg-zstream-upgrade-4-21-nh6gmd" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-flm2tp in resource group rg-zstream-upgrade-4-21-nh6gmd, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:141]: Unexpected error:
<*fmt.wrapErrors | 0xc0010545a0>:
failed to create HCP cluster cluster-zstream-4-21-flm2tp, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-flm2tp in resource group rg-zstream-upgrade-4-21-nh6gmd, error: failed waiting for cluster="cluster-zstream-4-21-flm2tp" in resourcegroup="rg-zstream-upgrade-4-21-nh6gmd" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-flm2tp in resource group rg-zstream-upgrade-4-21-nh6gmd, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
<*fmt.wrapErrors | 0xc00175a0c0>:
failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-b4dv64, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-b4dv64" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-b4dv64, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
<*fmt.wrapErrors | 0xc00175a0c0>:
failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-b4dv64, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-b4dv64" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-b4dv64, error: context deadline exceeded
...
occurredContributing tests (53)
Affected runs (4)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
context deadline exceededfull failure pattern: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 3 | 18.75%3 of 16 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsMay 10: 0 · May 11: 0 · May 12: 2 · May 13: 1 · May 14: 0 · May 15: 0 · May 16: 0 · May 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:132]: Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:132]: Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:139]: Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:139]: Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
...
occurredContributing tests (2)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Cluster should become ready within 45 minutesfull failure pattern: Cluster should become ready within 45 minutes Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | e2e | 2 | 12.50%2 of 16 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s) | Show trend detailsMay 10: 0 · May 11: 0 · May 12: 1 · May 13: 1 · May 14: 0 · May 15: 0 · May 16: 0 · May 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
<bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
<bool>: ...
to be trueContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 12.50%2 of 16 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsMay 10: 0 · May 11: 0 · May 12: 0 · May 13: 0 · May 14: 1 · May 15: 0 · May 16: 1 · May 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001818270>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-bvsh6f" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:262] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001818270>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-bvsh6f" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:249] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001916060>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-pqf8vs" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:249] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc001916060>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-pqf8vs" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurredContributing tests (2)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the APIfull failure pattern: add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 6.25%1 of 16 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMay 10: 0 · May 11: 0 · May 12: 1 · May 13: 0 · May 14: 0 · May 15: 0 · May 16: 0 · May 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:130]: Unexpected error:
<*fmt.wrapErrors | 0xc000cf90b0>:
add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:130]: Unexpected error:
<*fmt.wrapErrors | 0xc000cf90b0>:
add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 6.25%1 of 16 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsMay 10: 0 · May 11: 0 · May 12: 0 · May 13: 0 · May 14: 0 · May 15: 0 · May 16: 0 · May 17: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: Unexpected error:
<*fmt.wrapError | 0xc000e16640>:
route host was never found: Get "https://agnhost-e2e-serving-app-9t7jv.apps.aro.k6j5t2w7r5a6y7c.e1hu.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: Unexpected error:
<*fmt.wrapError | 0xc000e16640>:
route host was never found: Get "https://agnhost-e2e-serving-app-9t7jv.apps.aro.k6j5t2w7r5a6y7c.e1hu.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceededfull failure pattern: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 6.25%1 of 16 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsMay 10: 0 · May 11: 0 · May 12: 1 · May 13: 0 · May 14: 0 · May 15: 0 · May 16: 0 · May 17: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc00224e4c8>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-xwvt26" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc00224e4c8>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-xwvt26" to finish creating: context deadline exceeded
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||