CIHealth
<
Single Day: Aug 5▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: STG

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
23:00:03 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000fd8420>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-p7npg92ddx4m" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a0729b30-0b22-49e3-805f-acf532cd3054
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a0729b30-0b22-49e3-805f-acf532cd3054",
      "name": "a0729b30-0b22-49e3-805f-acf532cd3054",
      "status": "Failed",
      "startTime": "2026-08-05T23:26:29.027786333Z",
      "endTime": "2026-08-05T23:50:29.922451032Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s0n6iovcvlkt9agtmggl0qa3l8afqkg still exists (deletion dispatched at 2026-08-05T23:26:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000fd8420>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-p7npg92ddx4m" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a0729b30-0b22-49e3-805f-acf532cd3054
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a0729b30-0b22-49e3-805f-acf532cd3054",
      "name": "a0729b30-0b22-49e3-805f-acf532cd3054",
      "status": "Failed",
      "startTime": "2026-08-05T23:26:29.027786333Z",
      "endTime": "2026-08-05T23:50:29.922451032Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s0n6iovcvlkt9agtmggl0qa3l8afqkg still exists (deletion dispatched at 2026-08-05T23:26:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
23:00:03 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc00268e3c0>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-r9d5fs57v7m5, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-r9d5fs57v7m5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-r9d5fs57v7m5, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapErrors | 0xc00268e3c0>: 
    failed to create HCP cluster private-kas, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-r9d5fs57v7m5, error: failed waiting for cluster="private-kas" in resourcegroup="private-kas-r9d5fs57v7m5" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPCluster20251223FromParam for cluster private-kas in resource group private-kas-r9d5fs57v7m5, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
09:25:03 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0016623a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-wddjjs" in resourcegroup="admin-credential-lifecycle-test-2cjl4hgjbxzd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66adb307-293e-42a5-9c87-495ac203b3ae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66adb307-293e-42a5-9c87-495ac203b3ae",
      "name": "66adb307-293e-42a5-9c87-495ac203b3ae",
      "status": "Failed",
      "startTime": "2026-08-05T10:00:59.463734847Z",
      "endTime": "2026-08-05T10:25:05.241705565Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s0bflive11oegd4h0qd4ogh1aabqgng still exists (deletion dispatched at 2026-08-05T10:01:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc0016623a8>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-wddjjs" in resourcegroup="admin-credential-lifecycle-test-2cjl4hgjbxzd" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66adb307-293e-42a5-9c87-495ac203b3ae
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/66adb307-293e-42a5-9c87-495ac203b3ae",
      "name": "66adb307-293e-42a5-9c87-495ac203b3ae",
      "status": "Failed",
      "startTime": "2026-08-05T10:00:59.463734847Z",
      "endTime": "2026-08-05T10:25:05.241705565Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s0bflive11oegd4h0qd4ogh1aabqgng still exists (deletion dispatched at 2026-08-05T10:01:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
09:25:03 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-7q4c88jh8t9s/cluster-candidate-4-22-lmwnq5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a92b00>: 
    failed to create HCP cluster cluster-candidate-4-22-lmwnq5: failed waiting for cluster="cluster-candidate-4-22-lmwnq5" in resourcegroup="rg-candidate-4-22-7q4c88jh8t9s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0fbf75bb-5e9d-4978-9631-825e28be408a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0fbf75bb-5e9d-4978-9631-825e28be408a",
      "name": "0fbf75bb-5e9d-4978-9631-825e28be408a",
      "status": "Failed",
      "startTime": "2026-08-05T09:46:42.016692331Z",
      "endTime": "2026-08-05T10:05:54.175998153Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:121]: HCP cluster rg-candidate-4-22-7q4c88jh8t9s/cluster-candidate-4-22-lmwnq5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a92b00>: 
    failed to create HCP cluster cluster-candidate-4-22-lmwnq5: failed waiting for cluster="cluster-candidate-4-22-lmwnq5" in resourcegroup="rg-candidate-4-22-7q4c88jh8t9s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0fbf75bb-5e9d-4978-9631-825e28be408a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/0fbf75bb-5e9d-4978-9631-825e28be408a",
      "name": "0fbf75bb-5e9d-4978-9631-825e28be408a",
      "status": "Failed",
      "startTime": "2026-08-05T09:46:42.016692331Z",
      "endTime": "2026-08-05T10:05:54.175998153Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a