CIHealth
<
Single Day: Jul 27▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: INT

Time (UTC)JobFailed atResultPRFailed testsDetails
branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
21:07:12 UTC · failed at e2e · test HCP Nodepools GPU instances creates and deletes a GPU nodepool in a single cluster · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:130]: failed to create GPU nodepool gpu-np-1 with VM size Standard_NC4as_T4_v3
Unexpected error:
    <*fmt.wrapError | 0xc000657cc0>: 
    failed to create NodePool gpu-np-1: failed waiting for nodepool="gpu-np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-ldc6p42cl5nn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78fd648a-f595-4ce5-8eb8-ad0b9a27df53
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78fd648a-f595-4ce5-8eb8-ad0b9a27df53",
      "name": "78fd648a-f595-4ce5-8eb8-ad0b9a27df53",
      "status": "Failed",
      "startTime": "2026-07-27T22:23:39.075545784Z",
      "endTime": "2026-07-27T22:42:42.156031569Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/gpu_nodepools_create_delete.go:130]: failed to create GPU nodepool gpu-np-1 with VM size Standard_NC4as_T4_v3
Unexpected error:
    <*fmt.wrapError | 0xc000657cc0>: 
    failed to create NodePool gpu-np-1: failed waiting for nodepool="gpu-np-1" for cluster "cluster-gpu-np" in resourcegroup="rg-gpu-nodepool-ldc6p42cl5nn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78fd648a-f595-4ce5-8eb8-ad0b9a27df53
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/78fd648a-f595-4ce5-8eb8-ad0b9a27df53",
      "name": "78fd648a-f595-4ce5-8eb8-ad0b9a27df53",
      "status": "Failed",
      "startTime": "2026-07-27T22:23:39.075545784Z",
      "endTime": "2026-07-27T22:42:42.156031569Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift
21:07:12 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:109]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d09cc0>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-m9pqpmcwjsmn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dff9ef22-78f2-42c2-9500-26e9a2816985
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dff9ef22-78f2-42c2-9500-26e9a2816985",
      "name": "dff9ef22-78f2-42c2-9500-26e9a2816985",
      "status": "Failed",
      "startTime": "2026-07-27T22:07:40.259656598Z",
      "endTime": "2026-07-27T22:26:42.652667167Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:109]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d09cc0>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-m9pqpmcwjsmn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dff9ef22-78f2-42c2-9500-26e9a2816985
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dff9ef22-78f2-42c2-9500-26e9a2816985",
      "name": "dff9ef22-78f2-42c2-9500-26e9a2816985",
      "status": "Failed",
      "startTime": "2026-07-27T22:07:40.259656598Z",
      "endTime": "2026-07-27T22:26:42.652667167Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e; [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
17:31:54 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc001100150>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-zstream-upgrade-4-22-nl67lcz6k8wj", caused by: failed waiting for hcpCluster="cluster-zstream-4-22-tmmcpp" in resourcegroup="rg-zstream-upgrade-4-22-nl67lcz6k8wj" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-zstream-4-22-tmmcpp in resource group rg-zstream-upgrade-4-22-nl67lcz6k8wj, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-zstream-4-22-tmmcpp" in resourcegroup="rg-zstream-upgrade-4-22-nl67lcz6k8wj" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-zstream-4-22-tmmcpp in resource group rg-zstream-upgrade-4-22-nl67lcz6k8wj, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc001100150>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-zstream-upgrade-4-22-nl67lcz6k8wj", caused by: failed waiting for hcpCluster="cluster-zstream-4-22-tmmcpp" in resourcegroup="rg-zstream-upgrade-4-22-nl67lcz6k8wj" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-zstream-4-22-tmmcpp in resource group rg-zstream-upgrade-4-22-nl67lcz6k8wj, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-zstream-4-22-tmmcpp" in resourcegroup="rg-zstream-upgrade-4-22-nl67lcz6k8wj" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-zstream-4-22-tmmcpp in resource group rg-zstream-upgrade-4-22-nl67lcz6k8wj, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
tls: failed to verify certificate: x509: certificate signed by unknown authority
14:25:20 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:186]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-dqrl4d"
Unexpected error:
    <*fmt.wrapError | 0xc000ed73e0>: 
    [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-ch254.apps.aro.n0v1e5o4a9v5f2y.oymy.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:186]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-dqrl4d"
Unexpected error:
    <*fmt.wrapError | 0xc000ed73e0>: 
    [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-ch254.apps.aro.n0v1e5o4a9v5f2y.oymy.uksouth.aroapp-hcp.azure-test.net": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
list nodes (nodePool=<nodepool>): Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout
04:26:06 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:621]: failed to get node release images before downgrade
Unexpected error:
    <*fmt.wrapError | 0xc000cc1140>: 
    list nodes (nodePool=npdg-4-21): Get "https://api.r9l2c1j8d5l2w0y.xhnn.uksouth.aroapp-hcp.azure-test.net:443/api/v1/nodes": dial tcp: lookup api.r9l2c1j8d5l2w0y.xhnn.uksouth.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.129.184.62:37315->172.30.0.10:53: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:621]: failed to get node release images before downgrade
Unexpected error:
    <*fmt.wrapError | 0xc000cc1140>: 
    list nodes (nodePool=npdg-4-21): Get "https://api.r9l2c1j8d5l2w0y.xhnn.uksouth.aroapp-hcp.azure-test.net:443/api/v1/nodes": dial tcp: lookup api.r9l2c1j8d5l2w0y.xhnn.uksouth.aroapp-hcp.azure-test.net on 172.30.0.10:53: read udp 10.129.184.62:37315->172.30.0.10:53: i/o timeout
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
00:13:02 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite integration/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc00100e528>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-sww59j8gs8m4", caused by: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-sww59j8gs8m4" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-sww59j8gs8m4, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-sww59j8gs8m4" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-sww59j8gs8m4, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc00100e528>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-sww59j8gs8m4", caused by: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-sww59j8gs8m4" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-sww59j8gs8m4, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-sww59j8gs8m4" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-sww59j8gs8m4, error: context deadline exceeded
    ...
occurred