No failure patterns match your search.
Environment: STG
Failure patterns: 14 · Failures matched: 85 · Total runs: 28 · Runs affected: 18
| Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs. | Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern). | Number of distinct job runs where this failure pattern was detected. | Percentage of all job runs in this environment affected by this failure pattern during the selected window. | Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate. | TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size. | Also inOther environments where the same failure pattern was also detected during the selected window. | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
timeout during CreateHCPClusterAndWait; context deadline exceededfull failure pattern: timeout during CreateHCPClusterAndWait; context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | e2e | 11 | 39.29%11 of 28 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsJul 5: 0 · Jul 6: 4 · Jul 7: 4 · Jul 8: 2 · Jul 9: 0 · Jul 10: 8 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
<*fmt.wrapErrors | 0xc000accde0>:
failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-92v58x, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-92v58x" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-92v58x, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
<*fmt.wrapErrors | 0xc000accde0>:
failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-92v58x, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-92v58x" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-92v58x, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-21-gtv29c/cluster-candidate-4-21-6hdcwz should provision
Unexpected error:
<*fmt.wrapErrors | 0xc001116420>:
failed to create HCP cluster cluster-candidate-4-21-6hdcwz, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-6hdcwz in resource group rg-candidate-4-21-gtv29c, error: failed waiting for cluster="cluster-candidate-4-21-6hdcwz" in resourcegroup="rg-candidate-4-21-gtv29c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-6hdcwz in resource group rg-candidate-4-21-gtv29c, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-21-gtv29c/cluster-candidate-4-21-6hdcwz should provision
Unexpected error:
<*fmt.wrapErrors | 0xc001116420>:
failed to create HCP cluster cluster-candidate-4-21-6hdcwz, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-6hdcwz in resource group rg-candidate-4-21-gtv29c, error: failed waiting for cluster="cluster-candidate-4-21-6hdcwz" in resourcegroup="rg-candidate-4-21-gtv29c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-6hdcwz in resource group rg-candidate-4-21-gtv29c, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:101]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapErrors | 0xc0028aa5a0>:
failed to create HCP cluster np-autoscale-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-whrdjm, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-whrdjm" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-whrdjm, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:101]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
<*fmt.wrapErrors | 0xc0028aa5a0>:
failed to create HCP cluster np-autoscale-cluster, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-whrdjm, error: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-whrdjm" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster np-autoscale-cluster in resource group np-autoscaling-whrdjm, error: context deadline exceeded
...
occurredContributing tests (14)
Affected runs (11)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceededfull failure pattern: CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | e2e | 10 | 35.71%10 of 28 job runs affected | FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days | Show trend detailsJul 5: 0 · Jul 6: 13 · Jul 7: 11 · Jul 8: 5 · Jul 9: 0 · Jul 10: 1 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:151]: failed to create node pool npupgrade-4-20 with version 4.20.28
Unexpected error:
<*fmt.wrapErrors | 0xc002592660>:
failed to create NodePool npupgrade-4-20, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-wjg86x-d529tv, error: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-wjg86x" in resourcegroup="rg-np-version-upgrade-wjg86x-d529tv" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:151]: failed to create node pool npupgrade-4-20 with version 4.20.28
Unexpected error:
<*fmt.wrapErrors | 0xc002592660>:
failed to create NodePool npupgrade-4-20, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool npupgrade-4-20 in resource group rg-np-version-upgrade-wjg86x-d529tv, error: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-wjg86x" in resourcegroup="rg-np-version-upgrade-wjg86x-d529tv" to finish creating: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapErrors | 0xc0022a58c0>:
failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-nl4k78, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-nl4k78" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapErrors | 0xc0022a58c0>:
failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-nl4k78, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-nl4k78" to finish creating: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapErrors | 0xc0027151a0>:
failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-plcsc4, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-plcsc4" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
<*fmt.wrapErrors | 0xc0027151a0>:
failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-plcsc4, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-plcsc4" to finish creating: context deadline exceeded
...
occurredContributing tests (15)
Affected runs (10)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceededfull failure pattern: DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | e2e | 3 | 10.71%3 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history | Show trend detailsJul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 1 · Jul 9: 0 · Jul 10: 5 · Jul 11: 0 · Jul 12: 2 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283]: Unexpected error:
<*errors.joinError | 0xc00108a450>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-candidate-4-22-p27992", caused by: failed waiting for hcpCluster="cluster-candidate-4-22-bf67pt" in resourcegroup="rg-candidate-4-22-p27992" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-22-bf67pt in resource group rg-candidate-4-22-p27992, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-candidate-4-22-bf67pt" in resourcegroup="rg-candidate-4-22-p27992" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-22-bf67pt in resource group rg-candidate-4-22-p27992, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283]: Unexpected error:
<*errors.joinError | 0xc00108a450>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-candidate-4-22-p27992", caused by: failed waiting for hcpCluster="cluster-candidate-4-22-bf67pt" in resourcegroup="rg-candidate-4-22-p27992" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-22-bf67pt in resource group rg-candidate-4-22-p27992, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-candidate-4-22-bf67pt" in resourcegroup="rg-candidate-4-22-p27992" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-22-bf67pt in resource group rg-candidate-4-22-p27992, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283]: Unexpected error:
<*errors.joinError | 0xc001b34240>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-np-labels-taints-n2l8g2", caused by: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-n2l8g2" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-n2l8g2, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-n2l8g2" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-n2l8g2, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283]: Unexpected error:
<*errors.joinError | 0xc001b34240>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-np-labels-taints-n2l8g2", caused by: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-n2l8g2" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-n2l8g2, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-n2l8g2" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-n2l8g2, error: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283]: Unexpected error:
<*errors.joinError | 0xc000fb8498>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-np-skip-minor-7wfv2w-wrn8zb", caused by: failed waiting for hcpCluster="np-skip-minor-7wfv2w" in resourcegroup="rg-np-skip-minor-7wfv2w-wrn8zb" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster np-skip-minor-7wfv2w in resource group rg-np-skip-minor-7wfv2w-wrn8zb, error: context deadline exceeded, error: failed waiting for hcpCluster="np-skip-minor-7wfv2w" in resourcegroup="rg-np-skip-minor-7wfv2w-wrn8zb" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster np-skip-minor-7wfv2w in resource group rg-np-skip-minor-7wfv2w-wrn8zb, error: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:283]: Unexpected error:
<*errors.joinError | 0xc000fb8498>:
failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-np-skip-minor-7wfv2w-wrn8zb", caused by: failed waiting for hcpCluster="np-skip-minor-7wfv2w" in resourcegroup="rg-np-skip-minor-7wfv2w-wrn8zb" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster np-skip-minor-7wfv2w in resource group rg-np-skip-minor-7wfv2w-wrn8zb, error: context deadline exceeded, error: failed waiting for hcpCluster="np-skip-minor-7wfv2w" in resourcegroup="rg-np-skip-minor-7wfv2w-wrn8zb" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster np-skip-minor-7wfv2w in resource group rg-np-skip-minor-7wfv2w-wrn8zb, error: context deadline exceeded
...
occurredContributing tests (8)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
tls: failed to verify certificate: x509: certificate signed by unknown authorityfull failure pattern: tls: failed to verify certificate: x509: certificate signed by unknown authority Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | e2e | 3 | 10.71%3 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s) | Show trend detailsJul 5: 0 · Jul 6: 0 · Jul 7: 1 · Jul 8: 0 · Jul 9: 0 · Jul 10: 1 · Jul 11: 1 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
<*errors.joinError | 0xc0025541f8>:
VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-k7x5x.apps.aro.cilium-cluster.omy1.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:194]: failed to run simple web app and connectivity check with cilium CNI
Unexpected error:
<*errors.joinError | 0xc0025541f8>:
VerifySimpleWebApp failed: route was never reachable: Get "https://agnhost-e2e-serving-app-k7x5x.apps.aro.cilium-cluster.omy1.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-vcwc6l"
Unexpected error:
<*fmt.wrapError | 0xc000f82580>:
route was never reachable: Get "https://agnhost-e2e-serving-app-4lc9p.apps.aro.c9l5k9x0w5n8t1j.s25a.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-vcwc6l"
Unexpected error:
<*fmt.wrapError | 0xc000f82580>:
route was never reachable: Get "https://agnhost-e2e-serving-app-4lc9p.apps.aro.c9l5k9x0w5n8t1j.s25a.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-xjp2wq"
Unexpected error:
<*fmt.wrapError | 0xc000f18280>:
route was never reachable: Get "https://agnhost-e2e-serving-app-2x2t8.apps.aro.a3a2u9b0b6c9e9l.h96i.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:172]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-xjp2wq"
Unexpected error:
<*fmt.wrapError | 0xc000f18280>:
route was never reachable: Get "https://agnhost-e2e-serving-app-2x2t8.apps.aro.a3a2u9b0b6c9e9l.h96i.uksouthstaging.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
...
occurredContributing tests (3)
Affected runs (3)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
client rate limiter Wait returned an error: context deadline exceededfull failure pattern: client rate limiter Wait returned an error: context deadline exceeded Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | e2e | 2 | 7.14%2 of 28 job runs affected | IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s) | Show trend detailsJul 5: 0 · Jul 6: 0 · Jul 7: 1 · Jul 8: 1 · Jul 9: 0 · Jul 10: 0 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc0027903a8>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-qt596f, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-qt596f" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc0027903a8>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-qt596f, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-qt596f" to finish creating: context deadline exceeded
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc0013f8420>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-5rpjlm, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-5rpjlm" to finish creating: context deadline exceeded
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
<*errors.joinError | 0xc0013f8420>:
VerifyNodesReady failed: no nodes found in the cluster
VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
failed to create NodePool cilium-np, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool cilium-np in resource group complex-cilium-kv-5rpjlm, error: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-5rpjlm" to finish creating: context deadline exceeded
...
occurredContributing tests (1)
Affected runs (2)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting...full failure pattern: failed to get latest install version for candidate channel: query candidate graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 5: 0 · Jul 6: 10 · Jul 7: 0 · Jul 8: 0 · Jul 9: 0 · Jul 10: 0 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers) fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: query candidate graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Contributing tests (10)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailablefull failure pattern: after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 5: 0 · Jul 6: 4 · Jul 7: 0 · Jul 8: 0 · Jul 9: 0 · Jul 10: 0 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (3)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:65]: failed to determine z-stream install version for minor 4.21
Unexpected error:
<*fmt.wrapError | 0xc000d62860>:
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:65]: failed to determine z-stream install version for minor 4.21
Unexpected error:
<*fmt.wrapError | 0xc000d62860>:
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:400]: failed to get latest version for nodepool minor 4.20
Unexpected error:
<*fmt.wrapError | 0xc0008085c0>:
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:400]: failed to get latest version for nodepool minor 4.20
Unexpected error:
<*fmt.wrapError | 0xc0008085c0>:
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:536]: failed to get versions for minor 4.21
Unexpected error:
<*fmt.wrapError | 0xc000e1c460>:
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:536]: failed to get versions for minor 4.21
Unexpected error:
<*fmt.wrapError | 0xc000e1c460>:
after 4 attempts: ResponseFailed: unexpected HTTP status: 503 Service Unavailable
...
occurredContributing tests (4)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for <ocp-channel> returned 503 Service Unavailable: <html>full failure pattern: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for <ocp-channel> returned 503 Service Unavailable: <html> Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 5: 0 · Jul 6: 3 · Jul 7: 0 · Jul 8: 0 · Jul 9: 0 · Jul 10: 0 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 503 Service Unavailable: <html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style type="text/css">
body {
font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
line-height: 1.66666667;
font-size: 16px;
color: #333;
background-color: #fff;
margin: 2em 1em;
}
h1 {
font-size: 28px;
font-weight: 400;
}
p {
margin: 0 0 10px;
}
.alert.alert-info {
background-color: #F0F0F0;
margin-top: 30px;
padding: 30px;
}
.alert p {
padding-left: 35px;
}
ul {
padding-left: 51px;
position: relative;
}
li {
font-size: 14px;
margin-bottom: 1em;
}
p.info {
position: relative;
font-size: 20px;
}
p.info:before, p.info:after {
content: "";
left: 0;
position: absolute;
top: 0;
}
p.info:before {
background: #0066CC;
border-radius: 16px;
color: #fff;
content: "i";
font: bold 16px/24px serif;
height: 24px;
left: 0px;
text-align: center;
top: 4px;
width: 24px;
}
@media (min-width: 768px) {
body {
margin: 6em;
}
}
</style>
</head>
<body>
<div>
<h1>Application is not available</h1>
<p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>
<div class="alert alert-info">
<p class="info">
Possible reasons you are seeing this page:
</p>
<ul>
<li>
<strong>The host doesn't exist.</strong>
Make sure the hostname was typed correctly and that a route matching this hostname exists.
</li>
<li>
<strong>The host exists, but doesn't have a matching path.</strong>
Check if the URL path was typed correctly and that the route was created using the desired path.
</li>
<li>
<strong>Route and path matches, but all pods are down.</strong>
Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
</li>
</ul>
</div>
</div>
</body>
</html>
fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 503 Service Unavailable: <html>
<head>
<meta name="viewport" content="width=device-width, initial-scale=1">
<style type="text/css">
body {
font-family: "Helvetica Neue", Helvetica, Arial, sans-serif;
line-height: 1.66666667;
font-size: 16px;
color: #333;
background-color: #fff;
margin: 2em 1em;
}
h1 {
font-size: 28px;
font-weight: 400;
}
p {
margin: 0 0 10px;
}
.alert.alert-info {
background-color: #F0F0F0;
margin-top: 30px;
padding: 30px;
}
.alert p {
padding-left: 35px;
}
ul {
padding-left: 51px;
position: relative;
}
li {
font-size: 14px;
margin-bottom: 1em;
}
p.info {
position: relative;
font-size: 20px;
}
p.info:before, p.info:after {
content: "";
left: 0;
position: absolute;
top: 0;
}
p.info:before {
background: #0066CC;
border-radius: 16px;
color: #fff;
content: "i";
font: bold 16px/24px serif;
height: 24px;
left: 0px;
text-align: center;
top: 4px;
width: 24px;
}
@media (min-width: 768px) {
body {
margin: 6em;
}
}
</style>
</head>
<body>
<div>
<h1>Application is not available</h1>
<p>The application is currently not serving requests at this endpoint. It may not have been started or is still starting.</p>
<div class="alert alert-info">
<p class="info">
Possible reasons you are seeing this page:
</p>
<ul>
<li>
<strong>The host doesn't exist.</strong>
Make sure the hostname was typed correctly and that a route matching this hostname exists.
</li>
<li>
<strong>The host exists, but doesn't have a matching path.</strong>
Check if the URL path was typed correctly and that the route was created using the desired path.
</li>
<li>
<strong>Route and path matches, but all pods are down.</strong>
Make sure that the resources exposed by this route (pods, services, deployment configs, etc) have at least one pod running.
</li>
</ul>
</div>
</div>
</body>
</html>Contributing tests (3)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeoutfull failure pattern: after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 5: 0 · Jul 6: 2 · Jul 7: 0 · Jul 8: 0 · Jul 9: 0 · Jul 10: 0 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (2)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:677]: failed to get latest version for minor 4.21
Unexpected error:
<*fmt.wrapError | 0xc0010be160>:
after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:677]: failed to get latest version for minor 4.21
Unexpected error:
<*fmt.wrapError | 0xc0010be160>:
after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout
...
occurredfail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:65]: failed to determine z-stream install version for minor 4.20
Unexpected error:
<*fmt.wrapError | 0xc000c2e120>:
after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:65]: failed to determine z-stream install version for minor 4.20
Unexpected error:
<*fmt.wrapError | 0xc000c2e120>:
after 4 attempts: ResponseFailed: unexpected HTTP status: 504 Gateway Timeout
...
occurredContributing tests (2)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
ERROR CODE: NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/e2e-oidc-wi...full failure pattern: ERROR CODE: NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/e2e-oidc-wi-test' was not found.; provider Microsoft.ManagedIdentity Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 0 · Jul 9: 0 · Jul 10: 0 · Jul 11: 0 · Jul 12: 1 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:296]: failed to create federated identity credential for service account system:serviceaccount:e2e-oidc-wi:wi-test-sa
Unexpected error:
<*exported.ResponseError | 0xc0024efa70>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-m6pkth/providers/Microsoft.ManagedIdentity/userAssignedIdentities/e2e-oidc-wi-test/federatedIdentityCredentials/e2e-wi-fic
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/oidc-wi-m6pkth/providers/Microsoft.ManagedIdentity/userAssignedIdentities/e2e-oidc-wi-test' was not found."
}
}
--------------------------------------------------------------------------------
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:296]: failed to create federated identity credential for service account system:serviceaccount:e2e-oidc-wi:wi-test-sa
Unexpected error:
<*exported.ResponseError | 0xc0024efa70>:
PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-m6pkth/providers/Microsoft.ManagedIdentity/userAssignedIdentities/e2e-oidc-wi-test/federatedIdentityCredentials/e2e-wi-fic
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: NotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "NotFound",
"message": "Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/oidc-wi-m6pkth/providers/Microsoft.ManagedIdentity/userAssignedIdentities/e2e-oidc-wi-test' was not found."
}
}
--------------------------------------------------------------------------------
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to create FIPS check pod: pod "<pod>" is forbidden: error looking up service account <namespace>/default...full failure pattern: VerifyFIPSEnabled failed: failed to check FIPS mode on node <node>: failed to create FIPS check pod: pod "<pod>" is forbidden: error looking up service account <namespace>/default: serviceaccount "default" not found Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 1 · Jul 9: 0 · Jul 10: 0 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
<*errors.joinError | 0xc00107bab8>:
VerifyFIPSEnabled failed: failed to check FIPS mode on node x6i8r8c0s5b9y1e-np-1-xdg6k-5zg5s: failed to create FIPS check pod: pods "fips-check-" is forbidden: error looking up service account e2e-fips-test-qvwcn/default: serviceaccount "default" not found
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:123]: failed to verify FIPS is enabled on cluster fips-enabled-cluster
Unexpected error:
<*errors.joinError | 0xc00107bab8>:
VerifyFIPSEnabled failed: failed to check FIPS mode on node x6i8r8c0s5b9y1e-np-1-xdg6k-5zg5s: failed to create FIPS check pod: pods "fips-check-" is forbidden: error looking up service account e2e-fips-test-qvwcn/default: serviceaccount "default" not found
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for <ocp-channel> returned 504 Gateway Timeout: upstream request timeoutfull failure pattern: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for <ocp-channel> returned 504 Gateway Timeout: upstream request timeout Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 5: 0 · Jul 6: 1 · Jul 7: 0 · Jul 8: 0 · Jul 9: 0 · Jul 10: 0 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 504 Gateway Timeout: upstream request timeout fail [github.com/Azure/ARO-HCP/test/util/framework/deployment_params.go:179]: failed to get latest install version for candidate channel: after 4 attempts: query candidate graph for candidate-4.20 returned 504 Gateway Timeout: upstream request timeout Contributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish getting creds: Get "<url>": stream error: stream ID 175; CANCEL; received from peerfull failure pattern: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish getting creds: Get "<url>": stream error: stream ID 175; CANCEL; received from peer Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 5: 0 · Jul 6: 0 · Jul 7: 0 · Jul 8: 1 · Jul 9: 0 · Jul 10: 0 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:119]: failed to get admin REST config for cluster "cluster-zstream-4-22-dxshzj"
Unexpected error:
<*fmt.wrapError | 0xc0005104c0>:
failed waiting for hcpCluster="cluster-zstream-4-22-dxshzj" in resourcegroup="rg-zstream-upgrade-4-22-9npmw6" to finish getting creds: Get "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b1e7606d-ac0b-4b22-9951-0d6139fef826?api-version=2024-06-10-preview&t=639190961451825049&c=MIIHkTCCBnmgAwIBAgIRAIphu0X3wTKY-ML3gzs6k8owDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UEAxMrQ0NNRSBHMSBUTFMgUlNBIDIwNDggU0hBMjU2IDIwNDkgRVVTMiBDQSAwMTAeFw0yNjA0MDUxOTUxNDZaFw0yNjEwMDEwMTUxNDZaMEAxPjA8BgNVBAMTNWFzeW5jb3BlcmF0aW9uc2lnbmluZ2NlcnRpZmljYXRlLm1hbmFnZW1lbnQuYXp1cmUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnR-Iz8Fgy9ddzo1kN9_c9HX8fqbMo-rawsyhsE9PSveyJv-L6NSkYs-1wxuKxocBLyNq8Jah1OkhbHL2yHbaT2kH5ZLNyn03w4Fn5lmjGi0KRYdPQwRIoK_PTIYI3maaXA7Kya68KMyhJgqlwqg9IsBK3Dq2D3vr1XP-PogUr3cssjfqfW_pVnR8bWGDNVCFKHbZbXMBWTNpAW91fHtf-4ojMqnePnvHVqONRoxeIQxsLtx3bQSh4Hw2QF_n1pMCUJ52n2c3_kjlULgAIesOTCdL9Pqul7Qby0zkLW0k9ppiEtPlfX0NNTyex0tR5-UE6EfjaiZZq9c_3Er-iDFcyQIDAQABo4IEjjCCBIowgZ0GA1UdIASBlTCBkjAMBgorBgEEAYI3ewEBMGYGCisGAQQBgjd7AgIwWDBWBggrBgEFBQcCAjBKHkgAMwAzAGUAMAAxADkAMgAxAC0ANABkADYANAAtADQAZgA4AGMALQBhADAANQA1AC0ANQBiAGQAYQBmAGYAZAA1AGUAMwAzAGQwDAYKKwYBBAGCN3sDAjAMBgorBgEEAYI3ewQCMAwGA1UdEwEB_wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA4GA1UdDwEB_wQEAwIFoDAdBgNVHQ4EFgQU3jhSRQ1zlQcx7waACuHPEhCucfswHwYDVR0jBBgwFoAU_Ow-26p8H4IeBbihBvlD5wKzCrkwggGuBgNVHR8EggGlMIIBoTBooGagZIZiaHR0cDovL3ByaW1hcnktY2RuLnBraS5jb3JlLndpbmRvd3MubmV0L2Vhc3R1czIvY3Jscy9jY21lZWFzdHVzMnBraS9jY21lZWFzdHVzMmljYTAxLzMvY3VycmVudC5jcmwwaqBooGaGZGh0dHA6Ly9zZWNvbmRhcnktY2RuLnBraS5jb3JlLndpbmRvd3MubmV0L2Vhc3R1czIvY3Jscy9jY21lZWFzdHVzMnBraS9jY21lZWFzdHVzMmljYTAxLzMvY3VycmVudC5jcmwwWaBXoFWGU2h0dHA6Ly9jcmwubWljcm9zb2Z0LmNvbS9lYXN0dXMyL2NybHMvY2NtZWVhc3R1czJwa2kvY2NtZWVhc3R1czJpY2EwMS8zL2N1cnJlbnQuY3JsMG6gbKBqhmhodHRwOi8vY2NtZWVhc3R1czJwa2kuZWFzdHVzMi5wa2kuY29yZS53aW5kb3dzLm5ldC9jZXJ0aWZpY2F0ZUF1dGhvcml0aWVzL2NjbWVlYXN0dXMyaWNhMDEvMy9jdXJyZW50LmNybDCCAbcGCCsGAQUFBwEBBIIBqTCCAaUwbAYIKwYBBQUHMAKGYGh0dHA6Ly9wcmltYXJ5LWNkbi5wa2kuY29yZS53aW5kb3dzLm5ldC9lYXN0dXMyL2NhY2VydHMvY2NtZWVhc3R1czJwa2kvY2NtZWVhc3R1czJpY2EwMS9jZXJ0LmNlcjBuBggrBgEFBQcwAoZiaHR0cDovL3NlY29uZGFyeS1jZG4ucGtpLmNvcmUud2luZG93cy5uZXQvZWFzdHVzMi9jYWNlcnRzL2NjbWVlYXN0dXMycGtpL2NjbWVlYXN0dXMyaWNhMDEvY2VydC5jZXIwXQYIKwYBBQUHMAKGUWh0dHA6Ly9jcmwubWljcm9zb2Z0LmNvbS9lYXN0dXMyL2NhY2VydHMvY2NtZWVhc3R1czJwa2kvY2NtZWVhc3R1czJpY2EwMS9jZXJ0LmNlcjBmBggrBgEFBQcwAoZaaHR0cDovL2NjbWVlYXN0dXMycGtpLmVhc3R1czIucGtpLmNvcmUud2luZG93cy5uZXQvY2VydGlmaWNhdGVBdXRob3JpdGllcy9jY21lZWFzdHVzMmljYTAxMA0GCSqGSIb3DQEBCwUAA4IBAQAnTNmUYMgtA5djeWjFRnP8bzb2RSij05KzI6-zmrz8PwFhloqF_K-ppJyIpYnHi-6ahqLnr9NDS-inELxBS21PcqBnDx4YKiKbvb4iFBRzM1_p-Qs_Ohg8kxB7_s3yJJ42MBeISb04d8hwwG0Hzy6JDsvrcnavrUJxzl7SoGQI3cdVs3L3rOxBKrmixXdFJBIOLZeN8ZRU9cs7iMLzt3XqSfAQv2DpEbhLst8W5F29nJyo_XsgU4CHTh6VDbniACMzAYRvF-MXIf2gLnjEuX4FhnaYly_84KVjx8sr1f0nqFtfDL6X6yYAtrzSZb5lv2fBipHiLwLRt3loXadj5jPy&s=b5ROMIPn-mzkFhZhx7NJy1eqc88KFf26H3WdYCh2E00wIcC9xmZXEROuJk7fudR8yGlPltARYfDbKOKpcSYkLA5A6gwCpT2pH4NsmupfNRPd-P8xe0J2m45AxWZR-WtcVBKv3VNxXAbyTaJSo84LXGM7ST5afJo5LNpKcayqTDk9axgkBPzAHyy-h9WqL7x--k2U13Ypz5eZmoxTZai4_T0miL8G0wDuaCV8LHFDCSWgs-36AXlsHqQxcxKfPIivGQGUbs1cJmW8gkDwi3Ros2mPJz455jFhvHTweA_otcCI4aTcEPscwe91BPfUjtVgAnLzQJnm6_hBdI4X7qPt-g&h=H7Zn71yX1kKnQgEZVybYJDSWtRW2B5SVmIDaVUuNXsQ": stream error: stream ID 175; CANCEL; received from peer
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:119]: failed to get admin REST config for cluster "cluster-zstream-4-22-dxshzj"
Unexpected error:
<*fmt.wrapError | 0xc0005104c0>:
failed waiting for hcpCluster="cluster-zstream-4-22-dxshzj" in resourcegroup="rg-zstream-upgrade-4-22-9npmw6" to finish getting creds: Get "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b1e7606d-ac0b-4b22-9951-0d6139fef826?api-version=2024-06-10-preview&t=639190961451825049&c=MIIHkTCCBnmgAwIBAgIRAIphu0X3wTKY-ML3gzs6k8owDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UEAxMrQ0NNRSBHMSBUTFMgUlNBIDIwNDggU0hBMjU2IDIwNDkgRVVTMiBDQSAwMTAeFw0yNjA0MDUxOTUxNDZaFw0yNjEwMDEwMTUxNDZaMEAxPjA8BgNVBAMTNWFzeW5jb3BlcmF0aW9uc2lnbmluZ2NlcnRpZmljYXRlLm1hbmFnZW1lbnQuYXp1cmUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAnR-Iz8Fgy9ddzo1kN9_c9HX8fqbMo-rawsyhsE9PSveyJv-L6NSkYs-1wxuKxocBLyNq8Jah1OkhbHL2yHbaT2kH5ZLNyn03w4Fn5lmjGi0KRYdPQwRIoK_PTIYI3maaXA7Kya68KMyhJgqlwqg9IsBK3Dq2D3vr1XP-PogUr3cssjfqfW_pVnR8bWGDNVCFKHbZbXMBWTNpAW91fHtf-4ojMqnePnvHVqONRoxeIQxsLtx3bQSh4Hw2QF_n1pMCUJ52n2c3_kjlULgAIesOTCdL9Pqul7Qby0zkLW0k9ppiEtPlfX0NNTyex0tR5-UE6EfjaiZZq9c_3Er-iDFcyQIDAQABo4IEjjCCBIowgZ0GA1UdIASBlTCBkjAMBgorBgEEAYI3ewEBMGYGCisGAQQBgjd7AgIwWDBWBggrBgEFBQcCAjBKHkgAMwAzAGUAMAAxADkAMgAxAC0ANABkADYANAAtADQAZgA4AGMALQBhADAANQA1AC0ANQBiAGQAYQBmAGYAZAA1AGUAMwAzAGQwDAYKKwYBBAGCN3sDAjAMBgorBgEEAYI3ewQCMAwGA1UdEwEB_wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA4GA1UdDwEB_wQEAwIFoDAdBgNVHQ4EFgQU3jhSRQ1zlQcx7waACuHPEhCucfswHwYDVR0jBBgwFoAU_Ow-26p8H4IeBbihBvlD5wKzCrkwggGuBgNVHR8EggGlMIIBoTBooGagZIZiaHR0cDovL3ByaW1hcnktY2RuLnBraS5jb3JlLndpbmRvd3MubmV0L2Vhc3R1czIvY3Jscy9jY21lZWFzdHVzMnBraS9jY21lZWFzdHVzMmljYTAxLzMvY3VycmVudC5jcmwwaqBooGaGZGh0dHA6Ly9zZWNvbmRhcnktY2RuLnBraS5jb3JlLndpbmRvd3MubmV0L2Vhc3R1czIvY3Jscy9jY21lZWFzdHVzMnBraS9jY21lZWFzdHVzMmljYTAxLzMvY3VycmVudC5jcmwwWaBXoFWGU2h0dHA6Ly9jcmwubWljcm9zb2Z0LmNvbS9lYXN0dXMyL2NybHMvY2NtZWVhc3R1czJwa2kvY2NtZWVhc3R1czJpY2EwMS8zL2N1cnJlbnQuY3JsMG6gbKBqhmhodHRwOi8vY2NtZWVhc3R1czJwa2kuZWFzdHVzMi5wa2kuY29yZS53aW5kb3dzLm5ldC9jZXJ0aWZpY2F0ZUF1dGhvcml0aWVzL2NjbWVlYXN0dXMyaWNhMDEvMy9jdXJyZW50LmNybDCCAbcGCCsGAQUFBwEBBIIBqTCCAaUwbAYIKwYBBQUHMAKGYGh0dHA6Ly9wcmltYXJ5LWNkbi5wa2kuY29yZS53aW5kb3dzLm5ldC9lYXN0dXMyL2NhY2VydHMvY2NtZWVhc3R1czJwa2kvY2NtZWVhc3R1czJpY2EwMS9jZXJ0LmNlcjBuBggrBgEFBQcwAoZiaHR0cDovL3NlY29uZGFyeS1jZG4ucGtpLmNvcmUud2luZG93cy5uZXQvZWFzdHVzMi9jYWNlcnRzL2NjbWVlYXN0dXMycGtpL2NjbWVlYXN0dXMyaWNhMDEvY2VydC5jZXIwXQYIKwYBBQUHMAKGUWh0dHA6Ly9jcmwubWljcm9zb2Z0LmNvbS9lYXN0dXMyL2NhY2VydHMvY2NtZWVhc3R1czJwa2kvY2NtZWVhc3R1czJpY2EwMS9jZXJ0LmNlcjBmBggrBgEFBQcwAoZaaHR0cDovL2NjbWVlYXN0dXMycGtpLmVhc3R1czIucGtpLmNvcmUud2luZG93cy5uZXQvY2VydGlmaWNhdGVBdXRob3JpdGllcy9jY21lZWFzdHVzMmljYTAxMA0GCSqGSIb3DQEBCwUAA4IBAQAnTNmUYMgtA5djeWjFRnP8bzb2RSij05KzI6-zmrz8PwFhloqF_K-ppJyIpYnHi-6ahqLnr9NDS-inELxBS21PcqBnDx4YKiKbvb4iFBRzM1_p-Qs_Ohg8kxB7_s3yJJ42MBeISb04d8hwwG0Hzy6JDsvrcnavrUJxzl7SoGQI3cdVs3L3rOxBKrmixXdFJBIOLZeN8ZRU9cs7iMLzt3XqSfAQv2DpEbhLst8W5F29nJyo_XsgU4CHTh6VDbniACMzAYRvF-MXIf2gLnjEuX4FhnaYly_84KVjx8sr1f0nqFtfDL6X6yYAtrzSZb5lv2fBipHiLwLRt3loXadj5jPy&s=b5ROMIPn-mzkFhZhx7NJy1eqc88KFf26H3WdYCh2E00wIcC9xmZXEROuJk7fudR8yGlPltARYfDbKOKpcSYkLA5A6gwCpT2pH4NsmupfNRPd-P8xe0J2m45AxWZR-WtcVBKv3VNxXAbyTaJSo84LXGM7ST5afJo5LNpKcayqTDk9axgkBPzAHyy-h9WqL7x--k2U13Ypz5eZmoxTZai4_T0miL8G0wDuaCV8LHFDCSWgs-36AXlsHqQxcxKfPIivGQGUbs1cJmW8gkDwi3Ros2mPJz455jFhvHTweA_otcCI4aTcEPscwe91BPfUjtVgAnLzQJnm6_hBdI4X7qPt-g&h=H7Zn71yX1kKnQgEZVybYJDSWtRW2B5SVmIDaVUuNXsQ": stream error: stream ID 175; CANCEL; received from peer
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
query graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers)full failure pattern: query graph for <ocp-channel>: Get "<url>": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | e2e | 1 | 3.57%1 of 28 job runs affected | IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history | Show trend detailsJul 5: 0 · Jul 6: 1 · Jul 7: 0 · Jul 8: 0 · Jul 9: 0 · Jul 10: 0 · Jul 11: 0 · Jul 12: 0 | none | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Full failure examples (1)fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:257]: failed to get version pair without upgrade edge for minor 4.20
Unexpected error:
<*fmt.wrapError | 0xc00110cee0>:
query graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:257]: failed to get version pair without upgrade edge for minor 4.20
Unexpected error:
<*fmt.wrapError | 0xc00110cee0>:
query graph for candidate-4.20: Get "https://api.openshift.com/api/upgrades_info/v1/graph?channel=candidate-4.20": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
...
occurredContributing tests (1)
Affected runs (1)
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||