CIHealth
<
Single Day: Jul 28▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 4
e2e (4)
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
21:59:20 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f92ca0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-785v89f8zl6n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b678edbc-a8de-48c5-b4bf-98c9534b3023
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b678edbc-a8de-48c5-b4bf-98c9534b3023",
      "name": "b678edbc-a8de-48c5-b4bf-98c9534b3023",
      "status": "Failed",
      "startTime": "2026-07-28T22:17:57.975492057Z",
      "endTime": "2026-07-28T22:37:04.692327344Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f92ca0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-785v89f8zl6n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b678edbc-a8de-48c5-b4bf-98c9534b3023
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b678edbc-a8de-48c5-b4bf-98c9534b3023",
      "name": "b678edbc-a8de-48c5-b4bf-98c9534b3023",
      "status": "Failed",
      "startTime": "2026-07-28T22:17:57.975492057Z",
      "endTime": "2026-07-28T22:37:04.692327344Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
cluster entered terminal Failed state after role assignment deployment
21:59:20 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
21:59:20 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0006a57e0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-sctd8qw8snf8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a771521a-90f6-4fca-90a0-6d7413620b6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a771521a-90f6-4fca-90a0-6d7413620b6b",
      "name": "a771521a-90f6-4fca-90a0-6d7413620b6b",
      "status": "Failed",
      "startTime": "2026-07-28T22:17:53.088945399Z",
      "endTime": "2026-07-28T22:37:04.686017223Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0006a57e0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-sctd8qw8snf8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a771521a-90f6-4fca-90a0-6d7413620b6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a771521a-90f6-4fca-90a0-6d7413620b6b",
      "name": "a771521a-90f6-4fca-90a0-6d7413620b6b",
      "status": "Failed",
      "startTime": "2026-07-28T22:17:53.088945399Z",
      "endTime": "2026-07-28T22:37:04.686017223Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
21:59:20 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-21-jsj7cvtj96gz/cluster-candidate-4-21-b54lfg should provision
Unexpected error:
    <*fmt.wrapError | 0xc00061ab20>: 
    failed to create HCP cluster cluster-candidate-4-21-b54lfg: failed waiting for cluster="cluster-candidate-4-21-b54lfg" in resourcegroup="rg-candidate-4-21-jsj7cvtj96gz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2cfa579-4fda-40f6-ae95-536f1c6b4dac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2cfa579-4fda-40f6-ae95-536f1c6b4dac",
      "name": "c2cfa579-4fda-40f6-ae95-536f1c6b4dac",
      "status": "Failed",
      "startTime": "2026-07-28T22:18:57.247372966Z",
      "endTime": "2026-07-28T22:38:04.505191015Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-21-jsj7cvtj96gz/cluster-candidate-4-21-b54lfg should provision
Unexpected error:
    <*fmt.wrapError | 0xc00061ab20>: 
    failed to create HCP cluster cluster-candidate-4-21-b54lfg: failed waiting for cluster="cluster-candidate-4-21-b54lfg" in resourcegroup="rg-candidate-4-21-jsj7cvtj96gz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2cfa579-4fda-40f6-ae95-536f1c6b4dac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c2cfa579-4fda-40f6-ae95-536f1c6b4dac",
      "name": "c2cfa579-4fda-40f6-ae95-536f1c6b4dac",
      "status": "Failed",
      "startTime": "2026-07-28T22:18:57.247372966Z",
      "endTime": "2026-07-28T22:38:04.505191015Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
13:47:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-x876rf" for cluster "cluster-candidate-4-20-x876rf"
Unexpected error:
    <*fmt.wrapError | 0xc000a319c0>: 
    failed to create NodePool np-x876rf: failed waiting for nodepool="np-x876rf" for cluster "cluster-candidate-4-20-x876rf" in resourcegroup="rg-candidate-4-20-mgwk4gs6pnwj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8b082c6a-4522-4b1c-a483-2798c902dad7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8b082c6a-4522-4b1c-a483-2798c902dad7",
      "name": "8b082c6a-4522-4b1c-a483-2798c902dad7",
      "status": "Failed",
      "startTime": "2026-07-28T14:27:54.926915621Z",
      "endTime": "2026-07-28T14:47:02.131178681Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:173]: failed to create node pool "np-x876rf" for cluster "cluster-candidate-4-20-x876rf"
Unexpected error:
    <*fmt.wrapError | 0xc000a319c0>: 
    failed to create NodePool np-x876rf: failed waiting for nodepool="np-x876rf" for cluster "cluster-candidate-4-20-x876rf" in resourcegroup="rg-candidate-4-20-mgwk4gs6pnwj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8b082c6a-4522-4b1c-a483-2798c902dad7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8b082c6a-4522-4b1c-a483-2798c902dad7",
      "name": "8b082c6a-4522-4b1c-a483-2798c902dad7",
      "status": "Failed",
      "startTime": "2026-07-28T14:27:54.926915621Z",
      "endTime": "2026-07-28T14:47:02.131178681Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
tls: failed to verify certificate: x509: certificate signed by unknown authority
02:00:55 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:197]: failed to verify simple web app on back-level cluster version 4.19
Unexpected error:
    <*fmt.wrapError | 0xc000f24240>: 
    [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-4784h.apps.aro.x3i1t7t7q6r2c6g.uv5g.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:197]: failed to verify simple web app on back-level cluster version 4.19
Unexpected error:
    <*fmt.wrapError | 0xc000f24240>: 
    [strict TLS verification] route was never reachable: Get "https://agnhost-e2e-sample-app-4784h.apps.aro.x3i1t7t7q6r2c6g.uv5g.uksouth.aroapp-hcp.io": tls: failed to verify certificate: x509: certificate signed by unknown authority
    ...
occurred