CIHealth
<
Single Day: Aug 16▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: INT

Time (UTC)JobFailed atResultPRFailed testsDetails
branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
04:25:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000e21080>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-wxt5q8" in resourcegroup="rg-zstream-upgrade-4-21-2h58qhpsnrpc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/560bf569-795d-414b-a2b3-a9facd263303
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/560bf569-795d-414b-a2b3-a9facd263303",
      "name": "560bf569-795d-414b-a2b3-a9facd263303",
      "status": "Failed",
      "startTime": "2026-08-16T05:33:51.945312233Z",
      "endTime": "2026-08-16T05:57:52.72615778Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7fmjoq31pobk77og51cbq33r20i6c8 still exists (deletion dispatched at 2026-08-16T05:33:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000e21080>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-21-wxt5q8" in resourcegroup="rg-zstream-upgrade-4-21-2h58qhpsnrpc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/560bf569-795d-414b-a2b3-a9facd263303
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/560bf569-795d-414b-a2b3-a9facd263303",
      "name": "560bf569-795d-414b-a2b3-a9facd263303",
      "status": "Failed",
      "startTime": "2026-08-16T05:33:51.945312233Z",
      "endTime": "2026-08-16T05:57:52.72615778Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7fmjoq31pobk77og51cbq33r20i6c8 still exists (deletion dispatched at 2026-08-16T05:33:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 4
e2e (4)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
00:30:01 UTC · failed at e2e · test FIPS Mode Support with v2026-06-30 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000fe0258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-8fx5hz8rtgfc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f3328cc-bfbb-4690-87d9-7278e3150a65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f3328cc-bfbb-4690-87d9-7278e3150a65",
      "name": "9f3328cc-bfbb-4690-87d9-7278e3150a65",
      "status": "Failed",
      "startTime": "2026-08-16T01:09:17.93392394Z",
      "endTime": "2026-08-16T01:33:20.823057314Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7bmgbu2s9urdae5dap0s9uns7ic01k still exists (deletion dispatched at 2026-08-16T01:09:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000fe0258>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-8fx5hz8rtgfc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f3328cc-bfbb-4690-87d9-7278e3150a65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f3328cc-bfbb-4690-87d9-7278e3150a65",
      "name": "9f3328cc-bfbb-4690-87d9-7278e3150a65",
      "status": "Failed",
      "startTime": "2026-08-16T01:09:17.93392394Z",
      "endTime": "2026-08-16T01:33:20.823057314Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7bmgbu2s9urdae5dap0s9uns7ic01k still exists (deletion dispatched at 2026-08-16T01:09:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
00:30:01 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc002c10150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-vjqjrc66m7hp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef664610-d7a7-4211-b960-7c5ef19010a6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef664610-d7a7-4211-b960-7c5ef19010a6",
      "name": "ef664610-d7a7-4211-b960-7c5ef19010a6",
      "status": "Failed",
      "startTime": "2026-08-16T02:15:26.507725852Z",
      "endTime": "2026-08-16T02:39:32.627355563Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7ceegbsetggi2btn9o1aunt17miklq still exists (deletion dispatched at 2026-08-16T02:15:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc002c10150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-vjqjrc66m7hp" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef664610-d7a7-4211-b960-7c5ef19010a6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ef664610-d7a7-4211-b960-7c5ef19010a6",
      "name": "ef664610-d7a7-4211-b960-7c5ef19010a6",
      "status": "Failed",
      "startTime": "2026-08-16T02:15:26.507725852Z",
      "endTime": "2026-08-16T02:39:32.627355563Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7ceegbsetggi2btn9o1aunt17miklq still exists (deletion dispatched at 2026-08-16T02:15:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
00:30:01 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:122]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0xc000eb7820>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-lx86w6jrvwnc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34bc491d-3cad-42c9-9d5e-e1047be81dd6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34bc491d-3cad-42c9-9d5e-e1047be81dd6",
      "name": "34bc491d-3cad-42c9-9d5e-e1047be81dd6",
      "status": "Failed",
      "startTime": "2026-08-16T01:42:16.130592799Z",
      "endTime": "2026-08-16T02:01:21.628841591Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:122]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0xc000eb7820>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-lx86w6jrvwnc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34bc491d-3cad-42c9-9d5e-e1047be81dd6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34bc491d-3cad-42c9-9d5e-e1047be81dd6",
      "name": "34bc491d-3cad-42c9-9d5e-e1047be81dd6",
      "status": "Failed",
      "startTime": "2026-08-16T01:42:16.130592799Z",
      "endTime": "2026-08-16T02:01:21.628841591Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
00:30:01 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000cfc978>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-66rshlg99zgt" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ddf8e60-153e-4cf4-8998-18f59012fe70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ddf8e60-153e-4cf4-8998-18f59012fe70",
      "name": "8ddf8e60-153e-4cf4-8998-18f59012fe70",
      "status": "Failed",
      "startTime": "2026-08-16T01:41:05.507310234Z",
      "endTime": "2026-08-16T02:05:11.724659298Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7c9m5ga9c22o9l9gp6h377pus8tmsq still exists (deletion dispatched at 2026-08-16T01:41:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:293]: Unexpected error:
    <*errors.joinError | 0xc000cfc978>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-66rshlg99zgt" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ddf8e60-153e-4cf4-8998-18f59012fe70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ddf8e60-153e-4cf4-8998-18f59012fe70",
      "name": "8ddf8e60-153e-4cf4-8998-18f59012fe70",
      "status": "Failed",
      "startTime": "2026-08-16T01:41:05.507310234Z",
      "endTime": "2026-08-16T02:05:11.724659298Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2s7c9m5ga9c22o9l9gp6h377pus8tmsq still exists (deletion dispatched at 2026-08-16T01:41:05Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred