CIHealth
<
Single Day: Jul 23▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
23:00:47 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0001e0aa0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-9n27h62csfxn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/032e5270-928f-475d-9dae-95082a8a0af4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/032e5270-928f-475d-9dae-95082a8a0af4",
      "name": "032e5270-928f-475d-9dae-95082a8a0af4",
      "status": "Failed",
      "startTime": "2026-07-23T23:04:31.918481763Z",
      "endTime": "2026-07-23T23:23:43.694668207Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0001e0aa0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-9n27h62csfxn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/032e5270-928f-475d-9dae-95082a8a0af4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/032e5270-928f-475d-9dae-95082a8a0af4",
      "name": "032e5270-928f-475d-9dae-95082a8a0af4",
      "status": "Failed",
      "startTime": "2026-07-23T23:04:31.918481763Z",
      "endTime": "2026-07-23T23:23:43.694668207Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
cluster entered terminal Failed state after role assignment deployment
12:25:14 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
07:09:26 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00283c700>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-5676c2nc84jf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/81d6e351-7e5e-49b4-85a5-302b276cc8f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/81d6e351-7e5e-49b4-85a5-302b276cc8f6",
      "name": "81d6e351-7e5e-49b4-85a5-302b276cc8f6",
      "status": "Failed",
      "startTime": "2026-07-23T07:46:53.376501029Z",
      "endTime": "2026-07-23T08:05:58.659570015Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:136]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00283c700>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-5676c2nc84jf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/81d6e351-7e5e-49b4-85a5-302b276cc8f6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/81d6e351-7e5e-49b4-85a5-302b276cc8f6",
      "name": "81d6e351-7e5e-49b4-85a5-302b276cc8f6",
      "status": "Failed",
      "startTime": "2026-07-23T07:46:53.376501029Z",
      "endTime": "2026-07-23T08:05:58.659570015Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
07:09:26 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-hdmwqd" for cluster "cluster-candidate-4-20-hdmwqd"
Unexpected error:
    <*fmt.wrapError | 0xc001062200>: 
    failed to create NodePool np-hdmwqd: failed waiting for nodepool="np-hdmwqd" for cluster "cluster-candidate-4-20-hdmwqd" in resourcegroup="rg-candidate-4-20-grr9djf6qtvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c91f066e-3821-44ee-bece-edbab2bb31db
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c91f066e-3821-44ee-bece-edbab2bb31db",
      "name": "c91f066e-3821-44ee-bece-edbab2bb31db",
      "status": "Failed",
      "startTime": "2026-07-23T08:27:34.915224416Z",
      "endTime": "2026-07-23T08:46:40.245736142Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-hdmwqd" for cluster "cluster-candidate-4-20-hdmwqd"
Unexpected error:
    <*fmt.wrapError | 0xc001062200>: 
    failed to create NodePool np-hdmwqd: failed waiting for nodepool="np-hdmwqd" for cluster "cluster-candidate-4-20-hdmwqd" in resourcegroup="rg-candidate-4-20-grr9djf6qtvt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c91f066e-3821-44ee-bece-edbab2bb31db
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/c91f066e-3821-44ee-bece-edbab2bb31db",
      "name": "c91f066e-3821-44ee-bece-edbab2bb31db",
      "status": "Failed",
      "startTime": "2026-07-23T08:27:34.915224416Z",
      "endTime": "2026-07-23T08:46:40.245736142Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
03:27:16 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-hl8829" for cluster "cluster-candidate-4-21-hl8829"
Unexpected error:
    <*fmt.wrapError | 0xc0029b0ac0>: 
    failed to create NodePool np-hl8829: failed waiting for nodepool="np-hl8829" for cluster "cluster-candidate-4-21-hl8829" in resourcegroup="rg-candidate-4-21-tv28gtjd9f9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d4916d8a-cfac-4b24-ba00-5346b085835c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d4916d8a-cfac-4b24-ba00-5346b085835c",
      "name": "d4916d8a-cfac-4b24-ba00-5346b085835c",
      "status": "Failed",
      "startTime": "2026-07-23T04:16:36.200454364Z",
      "endTime": "2026-07-23T04:35:50.806314695Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-hl8829" for cluster "cluster-candidate-4-21-hl8829"
Unexpected error:
    <*fmt.wrapError | 0xc0029b0ac0>: 
    failed to create NodePool np-hl8829: failed waiting for nodepool="np-hl8829" for cluster "cluster-candidate-4-21-hl8829" in resourcegroup="rg-candidate-4-21-tv28gtjd9f9n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d4916d8a-cfac-4b24-ba00-5346b085835c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/d4916d8a-cfac-4b24-ba00-5346b085835c",
      "name": "d4916d8a-cfac-4b24-ba00-5346b085835c",
      "status": "Failed",
      "startTime": "2026-07-23T04:16:36.200454364Z",
      "endTime": "2026-07-23T04:35:50.806314695Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:46 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-s9bzk7" for cluster "cluster-candidate-4-20-s9bzk7"
Unexpected error:
    <*fmt.wrapError | 0xc0010305a0>: 
    failed to create NodePool np-s9bzk7: failed waiting for nodepool="np-s9bzk7" for cluster "cluster-candidate-4-20-s9bzk7" in resourcegroup="rg-candidate-4-20-l7l7vvhqm2rt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fabf9539-858a-402f-9f91-fefb74645581
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fabf9539-858a-402f-9f91-fefb74645581",
      "name": "fabf9539-858a-402f-9f91-fefb74645581",
      "status": "Failed",
      "startTime": "2026-07-23T02:18:43.115104063Z",
      "endTime": "2026-07-23T02:37:48.305364094Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-s9bzk7" for cluster "cluster-candidate-4-20-s9bzk7"
Unexpected error:
    <*fmt.wrapError | 0xc0010305a0>: 
    failed to create NodePool np-s9bzk7: failed waiting for nodepool="np-s9bzk7" for cluster "cluster-candidate-4-20-s9bzk7" in resourcegroup="rg-candidate-4-20-l7l7vvhqm2rt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fabf9539-858a-402f-9f91-fefb74645581
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/fabf9539-858a-402f-9f91-fefb74645581",
      "name": "fabf9539-858a-402f-9f91-fefb74645581",
      "status": "Failed",
      "startTime": "2026-07-23T02:18:43.115104063Z",
      "endTime": "2026-07-23T02:37:48.305364094Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
cluster entered terminal Failed state after role assignment deployment
02:00:46 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
add password after 2 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
00:28:28 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:132]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc0013a57d0>: 
    add password after 2 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:132]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc0013a57d0>: 
    add password after 2 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a