No failure patterns match your search.
Environment: PROD
Failure patterns: 8 · Failures matched: 82 · Total runs: 4 · Runs affected: 3
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days | e2e | 3 | 75.00%3 of 4 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days | Show trend detailsJun 17: 15 · Jun 18: 3 · Jun 19: 22 · Jun 20: 8 · Jun 21: 6 · Jun 22: 44 · Jun 23: 72 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-4-23-fm8zx8/cluster-nightly-4-23-jpcws7 should provision
Unexpected error:
<*fmt.wrapErrors | 0xc0000f6630>:
failed to create HCP cluster cluster-nightly-4-23-jpcws7, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-jpcws7 in resource group rg-nightly-4-23-fm8zx8, error: failed waiting for cluster="cluster-nightly-4-23-jpcws7" in resourcegroup="rg-nightly-4-23-fm8zx8" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-jpcws7 in resource group rg-nightly-4-23-fm8zx8, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-4-23-fm8zx8/cluster-nightly-4-23-jpcws7 should provision
Unexpected error:
<*fmt.wrapErrors | 0xc0000f6630>:
failed to create HCP cluster cluster-nightly-4-23-jpcws7, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-jpcws7 in resource group rg-nightly-4-23-fm8zx8, error: failed waiting for cluster="cluster-nightly-4-23-jpcws7" in resourcegroup="rg-nightly-4-23-fm8zx8" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-23-jpcws7 in resource group rg-nightly-4-23-fm8zx8, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
<*fmt.wrapErrors | 0xc000b8c6f0>:
failed to create HCP cluster cilium-cl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-8w8sfk, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-8w8sfk" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-8w8sfk, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
<*fmt.wrapErrors | 0xc000b8c6f0>:
failed to create HCP cluster cilium-cl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-8w8sfk, error: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-8w8sfk" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cilium-cl in resource group cni-cilium-8w8sfk, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:83]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapErrors | 0xc0014e0690>:
failed to create HCP cluster patch-name-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-hw4f87, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-hw4f87" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-hw4f87, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:83]: failed to create HCP cluster for patch-name test
Unexpected error:
<*fmt.wrapErrors | 0xc0014e0690>:
failed to create HCP cluster patch-name-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-hw4f87, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-hw4f87" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-hw4f87, error: context deadline exceeded
...
occurredContributing tests (43)
Affected runs (3)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Cluster should become ready within '20.000000' minutesfull failure pattern: Cluster should become ready within '20.000000' minutes Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | e2e | 2 | 50.00%2 of 4 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 3 days | Show trend detailsJun 17: 0 · Jun 18: 0 · Jun 19: 2 · Jun 20: 0 · Jun 21: 0 · Jun 22: 1 · Jun 23: 2 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.004s.
Cluster should become ready within '20.000000' minutes
Expected
<bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.004s.
Cluster should become ready within '20.000000' minutes
Expected
<bool>: ...
to be truefail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.001s.
Cluster should become ready within '20.000000' minutes
Expected
<bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:162]: Timed out after 1200.001s.
Cluster should become ready within '20.000000' minutes
Expected
<bool>: ...
to be trueContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
cluster has not yet reached Succeeded statefull failure pattern: cluster has not yet reached Succeeded state Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | e2e | 1 | 25.00%1 of 4 job runs affected | FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days | Show trend detailsJun 17: 1 · Jun 18: 0 · Jun 19: 2 · Jun 20: 0 · Jun 21: 0 · Jun 22: 1 · Jun 23: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
<armredhatopenshifthcp.ProvisioningState>: ...
to equal
<armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:199]: Timed out after 2520.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:197 with:
cluster has not yet reached Succeeded state
Expected
<armredhatopenshifthcp.ProvisioningState>: ...
to equal
<armredhatopenshifthcp.ProvisioningState>: ...Contributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 1 | 25.00%1 of 4 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsJun 17: 0 · Jun 18: 0 · Jun 19: 1 · Jun 20: 0 · Jun 21: 0 · Jun 22: 1 · Jun 23: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-bspklb"
Unexpected error:
<*fmt.wrapError | 0xc00061a820>:
route was never reachable: Get "https://agnhost-e2e-serving-app-2j48k.apps.aro.u6y1a6g9v3x2v7j.3efm.eastus2euap.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-bspklb"
Unexpected error:
<*fmt.wrapError | 0xc00061a820>:
route was never reachable: Get "https://agnhost-e2e-serving-app-2j48k.apps.aro.u6y1a6g9v3x2v7j.3efm.eastus2euap.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to wait for first cluster "<cluster>" to complete creation (timeout '20.000000' minutes)full failure pattern: failed to wait for first cluster "<cluster>" to complete creation (timeout '20.000000' minutes) Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 2 | 50.00%2 of 4 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsJun 17: 0 · Jun 18: 0 · Jun 19: 2 · Jun 20: 0 · Jun 21: 0 · Jun 22: 1 · Jun 23: 2 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '20.000000' minutes)
Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
...
occurredContributing tests (1)
Affected runs (2)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: c...full failure pattern: failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | e2e | 1 | 25.00%1 of 4 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s) | Show trend detailsJun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0 · Jun 22: 0 · Jun 23: 2 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:284] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0012842a0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-jmcbsh" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:284] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc0012842a0>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-jmcbsh" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:284] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00113d440>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-q78ff9" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:284] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
<*errors.joinError | 0xc00113d440>:
failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-q78ff9" to finish deleting: context canceled
...
occurred
fail [:0]: A node timeout occurredContributing tests (2)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | e2e | 1 | 25.00%1 of 4 job runs affected | IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s) | Show trend detailsJun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0 · Jun 22: 0 · Jun 23: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-wlsks5" for cluster "cluster-candidate-4-21-wlsks5"
Unexpected error:
<*fmt.wrapErrors | 0xc000a21440>:
failed to create NodePool np-wlsks5, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-wlsks5 in resource group rg-candidate-4-21-xnbmb4, error: failed waiting for nodepool="np-wlsks5" for cluster "cluster-candidate-4-21-wlsks5" in resourcegroup="rg-candidate-4-21-xnbmb4" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:159]: failed to create node pool "np-wlsks5" for cluster "cluster-candidate-4-21-wlsks5"
Unexpected error:
<*fmt.wrapErrors | 0xc000a21440>:
failed to create NodePool np-wlsks5, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-wlsks5 in resource group rg-candidate-4-21-xnbmb4, error: failed waiting for nodepool="np-wlsks5" for cluster "cluster-candidate-4-21-wlsks5" in resourcegroup="rg-candidate-4-21-xnbmb4" to finish creating: context deadline exceeded
...
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DeleteResourceGroup timed out after <minutes> minutes; context deadline exceededfull failure pattern: DeleteResourceGroup timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 25.00%1 of 4 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJun 17: 0 · Jun 18: 0 · Jun 19: 0 · Jun 20: 0 · Jun 21: 0 · Jun 22: 0 · Jun 23: 1 | none | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:145]: failed to delete customer resource group "cx-rg-cluster-lm6lx7"
Unexpected error:
<*fmt.wrapErrors | 0xc0015513e0>:
failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-lm6lx7, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:145]: failed to delete customer resource group "cx-rg-cluster-lm6lx7"
Unexpected error:
<*fmt.wrapErrors | 0xc0015513e0>:
failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-lm6lx7, error: context deadline exceeded
...
occurredContributing tests (1)
Affected runs (1)
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||