<
Single Day: Jul 25
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 7
e2e (7)
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
23:00:52 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc000ab02a0>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="private-keyvault-hzd8f756pkjx", caused by: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-hzd8f756pkjx" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster private-kv-cluster in resource group private-keyvault-hzd8f756pkjx, error: context deadline exceeded, error: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-hzd8f756pkjx" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster private-kv-cluster in resource group private-keyvault-hzd8f756pkjx, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc000ab02a0>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="private-keyvault-hzd8f756pkjx", caused by: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-hzd8f756pkjx" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster private-kv-cluster in resource group private-keyvault-hzd8f756pkjx, error: context deadline exceeded, error: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-hzd8f756pkjx" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster private-kv-cluster in resource group private-keyvault-hzd8f756pkjx, error: context deadline exceeded
    ...
occurred
DeleteResourceGroup timed out after <minutes> minutes; context deadline exceeded
23:00:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:145]: failed to delete customer resource group "cx-rg-cluster-jlqwkzs7jqhj"
Unexpected error:
    <*fmt.wrapErrors | 0xc001296270>: 
    failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-jlqwkzs7jqhj, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:145]: failed to delete customer resource group "cx-rg-cluster-jlqwkzs7jqhj"
Unexpected error:
    <*fmt.wrapErrors | 0xc001296270>: 
    failed to delete resource group, caused by: timeout '60.000000' minutes exceeded during DeleteResourceGroup for resource group cx-rg-cluster-jlqwkzs7jqhj, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
23:00:52 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-nightly-4-23-bvp8bvjdldql/cluster-nightly-4-23-6252tm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b819a0>: 
    failed to create HCP cluster cluster-nightly-4-23-6252tm: failed waiting for cluster="cluster-nightly-4-23-6252tm" in resourcegroup="rg-nightly-4-23-bvp8bvjdldql" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a07651a-a9b8-414c-8974-3522743736b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a07651a-a9b8-414c-8974-3522743736b5",
      "name": "7a07651a-a9b8-414c-8974-3522743736b5",
      "status": "Failed",
      "startTime": "2026-07-25T23:30:22.26941828Z",
      "endTime": "2026-07-25T23:49:39.833761299Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-nightly-4-23-bvp8bvjdldql/cluster-nightly-4-23-6252tm should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b819a0>: 
    failed to create HCP cluster cluster-nightly-4-23-6252tm: failed waiting for cluster="cluster-nightly-4-23-6252tm" in resourcegroup="rg-nightly-4-23-bvp8bvjdldql" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a07651a-a9b8-414c-8974-3522743736b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7a07651a-a9b8-414c-8974-3522743736b5",
      "name": "7a07651a-a9b8-414c-8974-3522743736b5",
      "status": "Failed",
      "startTime": "2026-07-25T23:30:22.26941828Z",
      "endTime": "2026-07-25T23:49:39.833761299Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster has no installed version; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
23:00:52 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc00172e378>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv", caused by: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-hszcm2jz5pbv, error: context deadline exceeded, error: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-hszcm2jz5pbv, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc00172e378>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv", caused by: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-hszcm2jz5pbv, error: context deadline exceeded, error: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hszcm2jz5pbv" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-hszcm2jz5pbv, error: context deadline exceeded
    ...
occurred
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
23:00:52 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc001028108>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-nightly-4-20-vdr9krj2r5gj", caused by: failed waiting for hcpCluster="cluster-nightly-4-20-2t8cfw" in resourcegroup="rg-nightly-4-20-vdr9krj2r5gj" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-nightly-4-20-2t8cfw in resource group rg-nightly-4-20-vdr9krj2r5gj, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-nightly-4-20-2t8cfw" in resourcegroup="rg-nightly-4-20-vdr9krj2r5gj" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-nightly-4-20-2t8cfw in resource group rg-nightly-4-20-vdr9krj2r5gj, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc001028108>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-nightly-4-20-vdr9krj2r5gj", caused by: failed waiting for hcpCluster="cluster-nightly-4-20-2t8cfw" in resourcegroup="rg-nightly-4-20-vdr9krj2r5gj" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-nightly-4-20-2t8cfw in resource group rg-nightly-4-20-vdr9krj2r5gj, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-nightly-4-20-2t8cfw" in resourcegroup="rg-nightly-4-20-vdr9krj2r5gj" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-nightly-4-20-2t8cfw in resource group rg-nightly-4-20-vdr9krj2r5gj, error: context deadline exceeded
    ...
occurred
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
23:00:52 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc00166a0d8>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="patch-name-xbgl6cqz2mnf", caused by: failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-xbgl6cqz2mnf" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster patch-name-cluster in resource group patch-name-xbgl6cqz2mnf, error: context deadline exceeded, error: failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-xbgl6cqz2mnf" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster patch-name-cluster in resource group patch-name-xbgl6cqz2mnf, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc00166a0d8>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="patch-name-xbgl6cqz2mnf", caused by: failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-xbgl6cqz2mnf" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster patch-name-cluster in resource group patch-name-xbgl6cqz2mnf, error: context deadline exceeded, error: failed waiting for hcpCluster="patch-name-cluster" in resourcegroup="patch-name-xbgl6cqz2mnf" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster patch-name-cluster in resource group patch-name-xbgl6cqz2mnf, error: context deadline exceeded
    ...
occurred
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
23:00:52 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc000fc0090>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="clusternp128-nzwfdf99d2bd", caused by: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-nzwfdf99d2bd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-nzwfdf99d2bd, error: context deadline exceeded, error: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-nzwfdf99d2bd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-nzwfdf99d2bd, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc000fc0090>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="clusternp128-nzwfdf99d2bd", caused by: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-nzwfdf99d2bd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-nzwfdf99d2bd, error: context deadline exceeded, error: failed waiting for hcpCluster="hcp-cluster-np-128" in resourcegroup="clusternp128-nzwfdf99d2bd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster hcp-cluster-np-128 in resource group clusternp128-nzwfdf99d2bd, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)