CIHealth
<
Single Day: Sep 17▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: STG

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
1h 2m 35s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
2h 12m 21s branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 1
e2e (1)
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "<url>": dial tcp: lookup <host> on <dns-server>: read udp <ip>-><ip>: i/o timeout
10:35:13 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:148]: failed to verify cluster "cp-ystream-upgrade-4-22-j5vkzr" is viable before upgrade
Unexpected error:
    <*errors.joinError | 0xc0015d2858>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": dial tcp: lookup api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io on 172.30.0.10:53: read udp 10.130.98.22:39829->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io on 172.30.0.10:53: read udp 10.130.98.22:39829->172.30.0.10:53: i/o timeout
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:148]: failed to verify cluster "cp-ystream-upgrade-4-22-j5vkzr" is viable before upgrade
Unexpected error:
    <*errors.joinError | 0xc0015d2858>: 
    VerifyBasicAccess failed: failed to list services: Get "https://api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io:443/api/v1/namespaces/default/services": dial tcp: lookup api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io on 172.30.0.10:53: read udp 10.130.98.22:39829->172.30.0.10:53: i/o timeout
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Get "https://api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io:443/apis/apiregistration.k8s.io/v1/apiservices": dial tcp: lookup api.z1t1t5s4i6u3r6h.s9td.uksouthstaging.aroapp-hcp.io on 172.30.0.10:53: read udp 10.130.98.22:39829->172.30.0.10:53: i/o timeout
    ...
occurred
2h 13m 11s branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 3
e2e (3)
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
05:11:15 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0035265a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-rcr2wwpjvcjv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b397553-ee61-4674-b26e-e3cf8ae2dc9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b397553-ee61-4674-b26e-e3cf8ae2dc9d",
      "name": "5b397553-ee61-4674-b26e-e3cf8ae2dc9d",
      "status": "Failed",
      "startTime": "2026-09-17T05:39:36.012772677Z",
      "endTime": "2026-09-17T05:58:42.168749593Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:139]: failed to create node pools in parallel for cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0035265a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-rcr2wwpjvcjv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b397553-ee61-4674-b26e-e3cf8ae2dc9d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5b397553-ee61-4674-b26e-e3cf8ae2dc9d",
      "name": "5b397553-ee61-4674-b26e-e3cf8ae2dc9d",
      "status": "Failed",
      "startTime": "2026-09-17T05:39:36.012772677Z",
      "endTime": "2026-09-17T05:58:42.168749593Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
05:11:15 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000fc2640>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-4xfvdn4xsffw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98b08bb9-0eee-4840-9535-91464d8458a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98b08bb9-0eee-4840-9535-91464d8458a2",
      "name": "98b08bb9-0eee-4840-9535-91464d8458a2",
      "status": "Failed",
      "startTime": "2026-09-17T05:40:34.02322608Z",
      "endTime": "2026-09-17T05:59:42.272250066Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000fc2640>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-4xfvdn4xsffw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98b08bb9-0eee-4840-9535-91464d8458a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98b08bb9-0eee-4840-9535-91464d8458a2",
      "name": "98b08bb9-0eee-4840-9535-91464d8458a2",
      "status": "Failed",
      "startTime": "2026-09-17T05:40:34.02322608Z",
      "endTime": "2026-09-17T05:59:42.272250066Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
05:11:15 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-72xhwq" for cluster "cluster-candidate-4-22-72xhwq"
Unexpected error:
    <*fmt.wrapError | 0xc0009bb400>: 
    failed to create NodePool np-72xhwq: failed waiting for nodepool="np-72xhwq" for cluster "cluster-candidate-4-22-72xhwq" in resourcegroup="rg-candidate-4-22-ffxlnxmpk4tb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ee3e506-9091-4aa0-ad2c-045d3e78731e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ee3e506-9091-4aa0-ad2c-045d3e78731e",
      "name": "8ee3e506-9091-4aa0-ad2c-045d3e78731e",
      "status": "Failed",
      "startTime": "2026-09-17T06:05:40.036081638Z",
      "endTime": "2026-09-17T06:24:44.492140979Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-72xhwq" for cluster "cluster-candidate-4-22-72xhwq"
Unexpected error:
    <*fmt.wrapError | 0xc0009bb400>: 
    failed to create NodePool np-72xhwq: failed waiting for nodepool="np-72xhwq" for cluster "cluster-candidate-4-22-72xhwq" in resourcegroup="rg-candidate-4-22-ffxlnxmpk4tb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ee3e506-9091-4aa0-ad2c-045d3e78731e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8ee3e506-9091-4aa0-ad2c-045d3e78731e",
      "name": "8ee3e506-9091-4aa0-ad2c-045d3e78731e",
      "status": "Failed",
      "startTime": "2026-09-17T06:05:40.036081638Z",
      "endTime": "2026-09-17T06:24:44.492140979Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
1h 55m 52s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth n/a passed n/a 0 n/a