<
Single Day: Aug 3
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: STG

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 2
e2e (2)
timeout during CreateHCPClusterAndWait; context deadline exceeded
23:00:01 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc0028142a0>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-gvvjrzgrplmx, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-gvvjrzgrplmx" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-gvvjrzgrplmx, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc0028142a0>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-gvvjrzgrplmx, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-gvvjrzgrplmx" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-gvvjrzgrplmx, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
23:00:01 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:328]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001248060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-n6mqxqbs7w4c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0a2b5ab8-4c39-4c7c-bc3d-5d14620f0783
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0a2b5ab8-4c39-4c7c-bc3d-5d14620f0783",
      "name": "0a2b5ab8-4c39-4c7c-bc3d-5d14620f0783",
      "status": "Failed",
      "startTime": "2026-08-03T23:17:59.310024619Z",
      "endTime": "2026-08-03T23:37:05.475161098Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:328]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001248060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-n6mqxqbs7w4c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0a2b5ab8-4c39-4c7c-bc3d-5d14620f0783
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0a2b5ab8-4c39-4c7c-bc3d-5d14620f0783",
      "name": "0a2b5ab8-4c39-4c7c-bc3d-5d14620f0783",
      "status": "Failed",
      "startTime": "2026-08-03T23:17:59.310024619Z",
      "endTime": "2026-08-03T23:37:05.475161098Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
e2e failed n/a 2
e2e (2)
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
14:03:21 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00166e138>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="arm64-vm-cluster-lz47frv6c784", caused by: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-lz47frv6c784" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lz47frv6c784, error: context deadline exceeded, error: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-lz47frv6c784" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lz47frv6c784, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc00166e138>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="arm64-vm-cluster-lz47frv6c784", caused by: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-lz47frv6c784" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lz47frv6c784, error: context deadline exceeded, error: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-lz47frv6c784" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-lz47frv6c784, error: context deadline exceeded
    ...
occurred
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
14:03:21 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011844b0>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="admin-credential-lifecycle-test-tc94pz2k8tgr", caused by: failed waiting for hcpCluster="admin-cred-lifecycle-d9x6dw" in resourcegroup="admin-credential-lifecycle-test-tc94pz2k8tgr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster admin-cred-lifecycle-d9x6dw in resource group admin-credential-lifecycle-test-tc94pz2k8tgr, error: context deadline exceeded, error: failed waiting for hcpCluster="admin-cred-lifecycle-d9x6dw" in resourcegroup="admin-credential-lifecycle-test-tc94pz2k8tgr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster admin-cred-lifecycle-d9x6dw in resource group admin-credential-lifecycle-test-tc94pz2k8tgr, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0011844b0>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="admin-credential-lifecycle-test-tc94pz2k8tgr", caused by: failed waiting for hcpCluster="admin-cred-lifecycle-d9x6dw" in resourcegroup="admin-credential-lifecycle-test-tc94pz2k8tgr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster admin-cred-lifecycle-d9x6dw in resource group admin-credential-lifecycle-test-tc94pz2k8tgr, error: context deadline exceeded, error: failed waiting for hcpCluster="admin-cred-lifecycle-d9x6dw" in resourcegroup="admin-credential-lifecycle-test-tc94pz2k8tgr" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster admin-cred-lifecycle-d9x6dw in resource group admin-credential-lifecycle-test-tc94pz2k8tgr, error: context deadline exceeded
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
n/a passed n/a 0 n/a