No failure patterns match your search.
Environment: PROD
Failure patterns: 7 · Failures matched: 23 · Total runs: 4 · Runs affected: 4
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host>full failure pattern: tls: failed to verify certificate: x509: certificate is valid for <hcp-api-host>, <hcp-api-host>, not <hcp-api-host> Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | e2e | 2 | 50.00%2 of 4 job runs affected | FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days | Show trend detailsApr 17: 0 · Apr 18: 0 · Apr 19: 0 · Apr 20: 3 · Apr 21: 0 · Apr 22: 3 · Apr 23: 9 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:161]: Unexpected error:
<*url.Error | 0xc00174e600>:
Get "https://api.t9i0s9f7m4g0f5e.0122.uksouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": tls: failed to verify certificate: x509: certificate is valid for api.t9i0s9f7m4g0f5e.rzw9.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t9i0s9f7m4g0f5e.0122.uksouth.aroapp-hcp.io
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:161]: Unexpected error:
<*url.Error | 0xc00174e600>:
Get "https://api.t9i0s9f7m4g0f5e.0122.uksouth.aroapp-hcp.io:443/apis/config.openshift.io/v1/clusterversions/version": tls: failed to verify certificate: x509: certificate is valid for api.t9i0s9f7m4g0f5e.rzw9.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t9i0s9f7m4g0f5e.0122.uksouth.aroapp-hcp.io
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: Unexpected error:
<*errors.joinError | 0xc000f427e0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
VerifyImageRegistryDisabled failed: wrong type of error: *url.Error, Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-image-registry/services/image-registry": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
VerifyBasicAccess failed: failed to list services: Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:108]: Unexpected error:
<*errors.joinError | 0xc000f427e0>:
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
VerifyImageRegistryDisabled failed: wrong type of error: *url.Error, Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/api/v1/namespaces/openshift-image-registry/services/image-registry": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
VerifyBasicAccess failed: failed to list services: Get "https://api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.t1y2o7l3w2e5r5q.viss.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.t1y2o7l3w2e5r5q.p2mn.uksouth.aroapp-hcp.io
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:160]: Unexpected error:
<*errors.joinError | 0xc00104ab10>:
VerifyBasicAccess failed: failed to list services: Get "https://api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.y8r9m4v2u9w1e2o.lmqw.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.y8r9m4v2u9w1e2o.lmqw.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:160]: Unexpected error:
<*errors.joinError | 0xc00104ab10>:
VerifyBasicAccess failed: failed to list services: Get "https://api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io:443/api/v1/namespaces/default/services": tls: failed to verify certificate: x509: certificate is valid for api.y8r9m4v2u9w1e2o.lmqw.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": tls: failed to verify certificate: x509: certificate is valid for api.y8r9m4v2u9w1e2o.lmqw.uksouth.aroapp-hcp.io, reserved.aroapp-hcp.io, not api.y8r9m4v2u9w1e2o.5d45.uksouth.aroapp-hcp.io
...
occurredContributing tests (8)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set: assertion failed: expected values to equalfull failure pattern: MISE Routing routes to the correct frontend based on version header MISE v2 when x-ms-mise-version header is set: assertion failed: expected values to equal Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 3 | 75.00%3 of 4 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsApr 17: 0 · Apr 18: 0 · Apr 19: 0 · Apr 20: 0 · Apr 21: 0 · Apr 22: 0 · Apr 23: 3 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
<string>: ...
to equal
<string>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/mise_routing.go:89]: Expected
<string>: ...
to equal
<string>: ...Contributing tests (1)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
client rate limiter Wait returned an error: context deadline exceededfull failure pattern: client rate limiter Wait returned an error: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | e2e | 2 | 50.00%2 of 4 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history | Show trend detailsApr 17: 0 · Apr 18: 0 · Apr 19: 0 · Apr 20: 0 · Apr 21: 0 · Apr 22: 4 · Apr 23: 2 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc000060d20>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-wt4d9m" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc000060d20>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-wt4d9m" to finish creating: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc001242648>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-qkvnrj" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: Unexpected error:
<*errors.joinError | 0xc001242648>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-qkvnrj" to finish creating: context deadline exceeded
...
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | e2e | 1 | 25.00%1 of 4 job runs affected | IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s) | Show trend detailsApr 17: 0 · Apr 18: 0 · Apr 19: 0 · Apr 20: 0 · Apr 21: 0 · Apr 22: 0 · Apr 23: 6 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapErrors | 0xc001592840>:
failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-rpl9lv, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-rpl9lv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-rpl9lv, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
<*fmt.wrapErrors | 0xc001592840>:
failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-rpl9lv, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-rpl9lv" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-rpl9lv, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
<*fmt.wrapErrors | 0xc000fd6d50>:
failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-qsxr69, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-qsxr69" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-qsxr69, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
<*fmt.wrapErrors | 0xc000fd6d50>:
failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-qsxr69, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-qsxr69" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-qsxr69, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
<*fmt.wrapErrors | 0xc0017b0570>:
failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-fsqjp5, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-fsqjp5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-fsqjp5, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
<*fmt.wrapErrors | 0xc0017b0570>:
failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-fsqjp5, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-fsqjp5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-fsqjp5, error: context deadline exceeded
...
occurredContributing tests (6)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
Cluster should become ready within 45 minutesfull failure pattern: Cluster should become ready within 45 minutes Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 25.00%1 of 4 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsApr 17: 0 · Apr 18: 0 · Apr 19: 0 · Apr 20: 0 · Apr 21: 0 · Apr 22: 0 · Apr 23: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
<bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
<bool>: ...
to be trueContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | e2e | 1 | 25.00%1 of 4 job runs affected | IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s) | Show trend detailsApr 17: 0 · Apr 18: 0 · Apr 19: 0 · Apr 20: 0 · Apr 21: 0 · Apr 22: 0 · Apr 23: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc0008ec4b0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-htjcmf, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-htjcmf" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:151]: Unexpected error:
<*fmt.wrapErrors | 0xc0008ec4b0>:
failed to create NodePool np-1, caused by: timeout '45.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-htjcmf, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-htjcmf" to finish creating: context deadline exceeded
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||
context deadline exceededfull failure pattern: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 25.00%1 of 4 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsApr 17: 0 · Apr 18: 0 · Apr 19: 0 · Apr 20: 0 · Apr 21: 0 · Apr 22: 0 · Apr 23: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: Unexpected error:
<context.deadlineExceededError>:
context deadline exceeded
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||