CIHealth
<
Single Day: Jun 22▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: STG

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 4
e2e (4)
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
23:00:05 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000012ba0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-kjlqnf" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc000012ba0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-kjlqnf" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
23:00:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-4-22-6f2d8n/cluster-nightly-4-22-vbj6q7 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0018126f0>: 
    failed to create HCP cluster cluster-nightly-4-22-vbj6q7, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-22-vbj6q7 in resource group rg-nightly-4-22-6f2d8n, error: failed waiting for cluster="cluster-nightly-4-22-vbj6q7" in resourcegroup="rg-nightly-4-22-6f2d8n" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-22-vbj6q7 in resource group rg-nightly-4-22-6f2d8n, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-nightly-4-22-6f2d8n/cluster-nightly-4-22-vbj6q7 should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0018126f0>: 
    failed to create HCP cluster cluster-nightly-4-22-vbj6q7, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-22-vbj6q7 in resource group rg-nightly-4-22-6f2d8n, error: failed waiting for cluster="cluster-nightly-4-22-vbj6q7" in resourcegroup="rg-nightly-4-22-6f2d8n" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-nightly-4-22-vbj6q7 in resource group rg-nightly-4-22-6f2d8n, error: context deadline exceeded
    ...
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
23:00:05 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001800348>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listzr65l9" in resourcegroup="ea-list-z9h2z8" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc001800348>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listzr65l9" in resourcegroup="ea-list-z9h2z8" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
CreateNodePoolFromParam timed out after <minutes> minutes; context deadline exceeded
23:00:05 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc0016aa030>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-6r4vst, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-6r4vst" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapErrors | 0xc0016aa030>: 
    failed to create NodePool np-1, caused by: timeout '20.000000' minutes exceeded during CreateNodePoolFromParam for node pool np-1 in resource group private-keyvault-6r4vst, error: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-6r4vst" to finish creating: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
timeout during CreateHCPClusterAndWait; context deadline exceeded
14:54:35 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-20-8n2xlf/cluster-candidate-4-20-rb4p4z should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0012ae630>: 
    failed to create HCP cluster cluster-candidate-4-20-rb4p4z, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: failed waiting for cluster="cluster-candidate-4-20-rb4p4z" in resourcegroup="rg-candidate-4-20-8n2xlf" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-20-8n2xlf/cluster-candidate-4-20-rb4p4z should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc0012ae630>: 
    failed to create HCP cluster cluster-candidate-4-20-rb4p4z, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: failed waiting for cluster="cluster-candidate-4-20-rb4p4z" in resourcegroup="rg-candidate-4-20-8n2xlf" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-20-rb4p4z in resource group rg-candidate-4-20-8n2xlf, error: context deadline exceeded
    ...
occurred
failed to cleanup resource group: at least one HCP cluster <cluster> to delete: failed waiting for hcpcluster="<cluster>" in resourcegroup="<resource-group>" to finish deleting: context canceled
14:54:35 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0010821f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-4s6fx4" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285] A node timeout occurred and then the following failure was recorded in the timedout node before it exited:
Unexpected error:
    <*errors.joinError | 0xc0010821f8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-4s6fx4" to finish deleting: context canceled
    ...
occurred  
fail [:0]: A node timeout occurred
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 3
e2e (3)
timeout during CreateHCPClusterAndWait; context deadline exceeded
10:20:59 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-21-gn2h29/cluster-candidate-4-21-jmhjfc should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001400a80>: 
    failed to create HCP cluster cluster-candidate-4-21-jmhjfc, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-jmhjfc in resource group rg-candidate-4-21-gn2h29, error: failed waiting for cluster="cluster-candidate-4-21-jmhjfc" in resourcegroup="rg-candidate-4-21-gn2h29" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-jmhjfc in resource group rg-candidate-4-21-gn2h29, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:106]: HCP cluster rg-candidate-4-21-gn2h29/cluster-candidate-4-21-jmhjfc should provision
Unexpected error:
    <*fmt.wrapErrors | 0xc001400a80>: 
    failed to create HCP cluster cluster-candidate-4-21-jmhjfc, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-jmhjfc in resource group rg-candidate-4-21-gn2h29, error: failed waiting for cluster="cluster-candidate-4-21-jmhjfc" in resourcegroup="rg-candidate-4-21-gn2h29" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster cluster-candidate-4-21-jmhjfc in resource group rg-candidate-4-21-gn2h29, error: context deadline exceeded
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
10:20:59 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-nhtnkl" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc0015a5aa0>: 
    failed to create HCP cluster cluster-zstream-4-22-nhtnkl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-nhtnkl in resource group rg-zstream-upgrade-4-22-jdkmgf, error: failed waiting for cluster="cluster-zstream-4-22-nhtnkl" in resourcegroup="rg-zstream-upgrade-4-22-jdkmgf" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-nhtnkl in resource group rg-zstream-upgrade-4-22-jdkmgf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-22-nhtnkl" with version 4.22.0 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc0015a5aa0>: 
    failed to create HCP cluster cluster-zstream-4-22-nhtnkl, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-nhtnkl in resource group rg-zstream-upgrade-4-22-jdkmgf, error: failed waiting for cluster="cluster-zstream-4-22-nhtnkl" in resourcegroup="rg-zstream-upgrade-4-22-jdkmgf" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-22-nhtnkl in resource group rg-zstream-upgrade-4-22-jdkmgf, error: context deadline exceeded
    ...
occurred
add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
10:20:59 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:269]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc000d45830>: 
    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:269]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc000d45830>: 
    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
timeout during CreateHCPClusterAndWait; context deadline exceeded
08:15:48 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-20-xb84n7" with version 4.20.25 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc000316570>: 
    failed to create HCP cluster cluster-zstream-4-20-xb84n7, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-xb84n7 in resource group rg-zstream-upgrade-4-20-fn7428, error: failed waiting for cluster="cluster-zstream-4-20-xb84n7" in resourcegroup="rg-zstream-upgrade-4-20-fn7428" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-xb84n7 in resource group rg-zstream-upgrade-4-20-fn7428, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:102]: failed to create HCP cluster "cluster-zstream-4-20-xb84n7" with version 4.20.25 on candidate channel
Unexpected error:
    <*fmt.wrapErrors | 0xc000316570>: 
    failed to create HCP cluster cluster-zstream-4-20-xb84n7, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-xb84n7 in resource group rg-zstream-upgrade-4-20-fn7428, error: failed waiting for cluster="cluster-zstream-4-20-xb84n7" in resourcegroup="rg-zstream-upgrade-4-20-fn7428" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-xb84n7 in resource group rg-zstream-upgrade-4-20-fn7428, error: context deadline exceeded
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
expected <count> ready (and schedulable) nodes, found <count>
02:00:28 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231]: failed to verify 3 ready and schedulable nodes for nodepool npupgrade-4-20 after upgrade
Expected success, but got an error:
    <*errors.errorString | 0xc000f85a50>: 
    expected 3 ready (and schedulable) nodes in node pool "npupgrade-4-20", found 4
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:231]: failed to verify 3 ready and schedulable nodes for nodepool npupgrade-4-20 after upgrade
Expected success, but got an error:
    <*errors.errorString | 0xc000f85a50>: 
    expected 3 ready (and schedulable) nodes in node pool "npupgrade-4-20", found 4
    ...