CIHealth
<
Single Day: Jul 23▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: STG

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 13
e2e (13)
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test FIPS Mode Support with ExperimentalReleaseFeatures AFEC registered should create an HCP cluster with FIPS mode enabled via experimental tag · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:84]: failed to create HCP cluster "fips-enabled-cluster" with FIPS enabled
Unexpected error:
    <*fmt.wrapError | 0xc000c4c4e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-z6d2ccxkwbm9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc7945ac-71c6-41ff-8506-5f96ae9eb9aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc7945ac-71c6-41ff-8506-5f96ae9eb9aa",
      "name": "dc7945ac-71c6-41ff-8506-5f96ae9eb9aa",
      "status": "Failed",
      "startTime": "2026-07-23T14:56:53.295655313Z",
      "endTime": "2026-07-23T15:16:00.70859883Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:84]: failed to create HCP cluster "fips-enabled-cluster" with FIPS enabled
Unexpected error:
    <*fmt.wrapError | 0xc000c4c4e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-z6d2ccxkwbm9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc7945ac-71c6-41ff-8506-5f96ae9eb9aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dc7945ac-71c6-41ff-8506-5f96ae9eb9aa",
      "name": "dc7945ac-71c6-41ff-8506-5f96ae9eb9aa",
      "status": "Failed",
      "startTime": "2026-07-23T14:56:53.295655313Z",
      "endTime": "2026-07-23T15:16:00.70859883Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:159]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000cc5380>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-fmnzp4sdw2lh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d433c12f-48e3-4234-8059-98785a534ca4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d433c12f-48e3-4234-8059-98785a534ca4",
      "name": "d433c12f-48e3-4234-8059-98785a534ca4",
      "status": "Failed",
      "startTime": "2026-07-23T14:56:00.123383638Z",
      "endTime": "2026-07-23T15:15:00.488210427Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:159]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000cc5380>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-fmnzp4sdw2lh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d433c12f-48e3-4234-8059-98785a534ca4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d433c12f-48e3-4234-8059-98785a534ca4",
      "name": "d433c12f-48e3-4234-8059-98785a534ca4",
      "status": "Failed",
      "startTime": "2026-07-23T14:56:00.123383638Z",
      "endTime": "2026-07-23T15:15:00.488210427Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:111]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000c5d0a0>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-kzmh2hsvhvk8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e6475fb8-7d66-4a3f-9a11-c8ce9a4ff903
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e6475fb8-7d66-4a3f-9a11-c8ce9a4ff903",
      "name": "e6475fb8-7d66-4a3f-9a11-c8ce9a4ff903",
      "status": "Failed",
      "startTime": "2026-07-23T15:38:51.52148988Z",
      "endTime": "2026-07-23T15:57:51.229369914Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:111]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000c5d0a0>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-kzmh2hsvhvk8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e6475fb8-7d66-4a3f-9a11-c8ce9a4ff903
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e6475fb8-7d66-4a3f-9a11-c8ce9a4ff903",
      "name": "e6475fb8-7d66-4a3f-9a11-c8ce9a4ff903",
      "status": "Failed",
      "startTime": "2026-07-23T15:38:51.52148988Z",
      "endTime": "2026-07-23T15:57:51.229369914Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc00107c1a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-k675rkhrbpwf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b7e2f89c-f64e-43f3-98b1-b2a73b98579c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b7e2f89c-f64e-43f3-98b1-b2a73b98579c",
      "name": "b7e2f89c-f64e-43f3-98b1-b2a73b98579c",
      "status": "Failed",
      "startTime": "2026-07-23T14:58:12.217171113Z",
      "endTime": "2026-07-23T15:17:30.750214168Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:97]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc00107c1a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-k675rkhrbpwf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b7e2f89c-f64e-43f3-98b1-b2a73b98579c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b7e2f89c-f64e-43f3-98b1-b2a73b98579c",
      "name": "b7e2f89c-f64e-43f3-98b1-b2a73b98579c",
      "status": "Failed",
      "startTime": "2026-07-23T14:58:12.217171113Z",
      "endTime": "2026-07-23T15:17:30.750214168Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc0008f41c0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-d7jl5l9g8t4g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4fca8419-704b-41d1-b240-785c6b2babf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4fca8419-704b-41d1-b240-785c6b2babf8",
      "name": "4fca8419-704b-41d1-b240-785c6b2babf8",
      "status": "Failed",
      "startTime": "2026-07-23T14:56:20.757280221Z",
      "endTime": "2026-07-23T15:15:30.708956769Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:83]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc0008f41c0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-d7jl5l9g8t4g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4fca8419-704b-41d1-b240-785c6b2babf8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/4fca8419-704b-41d1-b240-785c6b2babf8",
      "name": "4fca8419-704b-41d1-b240-785c6b2babf8",
      "status": "Failed",
      "startTime": "2026-07-23T14:56:20.757280221Z",
      "endTime": "2026-07-23T15:15:30.708956769Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-22-2hkmp8kgmll4/cluster-candidate-4-22-zqcp95 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c78320>: 
    failed to create HCP cluster cluster-candidate-4-22-zqcp95: failed waiting for cluster="cluster-candidate-4-22-zqcp95" in resourcegroup="rg-candidate-4-22-2hkmp8kgmll4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7dd0acb6-1bcd-4cbd-a77e-1e2698a80ca7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7dd0acb6-1bcd-4cbd-a77e-1e2698a80ca7",
      "name": "7dd0acb6-1bcd-4cbd-a77e-1e2698a80ca7",
      "status": "Failed",
      "startTime": "2026-07-23T14:57:03.082121392Z",
      "endTime": "2026-07-23T15:16:10.601844479Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-22-2hkmp8kgmll4/cluster-candidate-4-22-zqcp95 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c78320>: 
    failed to create HCP cluster cluster-candidate-4-22-zqcp95: failed waiting for cluster="cluster-candidate-4-22-zqcp95" in resourcegroup="rg-candidate-4-22-2hkmp8kgmll4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7dd0acb6-1bcd-4cbd-a77e-1e2698a80ca7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7dd0acb6-1bcd-4cbd-a77e-1e2698a80ca7",
      "name": "7dd0acb6-1bcd-4cbd-a77e-1e2698a80ca7",
      "status": "Failed",
      "startTime": "2026-07-23T14:57:03.082121392Z",
      "endTime": "2026-07-23T15:16:10.601844479Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 2 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
timeout during CreateHCPClusterAndWait; context deadline exceeded
14:34:14 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
    <*fmt.wrapErrors | 0xc000f326f0>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-qp8jkwnz8g6c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-qp8jkwnz8g6c, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: failed to create HCP cluster version 4.19
Unexpected error:
    <*fmt.wrapErrors | 0xc000f326f0>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-qp8jkwnz8g6c" to finish creating, caused by: timeout '20.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-qp8jkwnz8g6c, error: context deadline exceeded
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00045ef40>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-btrlf49mx4mr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f8bb69e-3dbc-46bc-b77f-ac4c9b9f8f96
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f8bb69e-3dbc-46bc-b77f-ac4c9b9f8f96",
      "name": "5f8bb69e-3dbc-46bc-b77f-ac4c9b9f8f96",
      "status": "Failed",
      "startTime": "2026-07-23T15:29:57.199203941Z",
      "endTime": "2026-07-23T15:49:11.470560955Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00045ef40>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-btrlf49mx4mr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f8bb69e-3dbc-46bc-b77f-ac4c9b9f8f96
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/5f8bb69e-3dbc-46bc-b77f-ac4c9b9f8f96",
      "name": "5f8bb69e-3dbc-46bc-b77f-ac4c9b9f8f96",
      "status": "Failed",
      "startTime": "2026-07-23T15:29:57.199203941Z",
      "endTime": "2026-07-23T15:49:11.470560955Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
cluster entered terminal Failed state after role assignment deployment
14:34:14 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-20-kr2pqb" with version 4.20.29 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000e02700>: 
    failed to create HCP cluster cluster-zstream-4-20-kr2pqb: failed waiting for cluster="cluster-zstream-4-20-kr2pqb" in resourcegroup="rg-zstream-upgrade-4-20-5gp2mtdbkppf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1b92cfe-14c4-4841-a691-dcb15fd4bada
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1b92cfe-14c4-4841-a691-dcb15fd4bada",
      "name": "a1b92cfe-14c4-4841-a691-dcb15fd4bada",
      "status": "Failed",
      "startTime": "2026-07-23T15:33:05.814258572Z",
      "endTime": "2026-07-23T15:52:11.319531705Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:109]: failed to create HCP cluster "cluster-zstream-4-20-kr2pqb" with version 4.20.29 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000e02700>: 
    failed to create HCP cluster cluster-zstream-4-20-kr2pqb: failed waiting for cluster="cluster-zstream-4-20-kr2pqb" in resourcegroup="rg-zstream-upgrade-4-20-5gp2mtdbkppf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1b92cfe-14c4-4841-a691-dcb15fd4bada
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a1b92cfe-14c4-4841-a691-dcb15fd4bada",
      "name": "a1b92cfe-14c4-4841-a691-dcb15fd4bada",
      "status": "Failed",
      "startTime": "2026-07-23T15:33:05.814258572Z",
      "endTime": "2026-07-23T15:52:11.319531705Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:269]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001012a40>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-vtmwlt59mcqb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98e7c2b5-126e-4d99-a480-38555e004a37
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98e7c2b5-126e-4d99-a480-38555e004a37",
      "name": "98e7c2b5-126e-4d99-a480-38555e004a37",
      "status": "Failed",
      "startTime": "2026-07-23T15:32:15.307228267Z",
      "endTime": "2026-07-23T15:51:31.367394587Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:269]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001012a40>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-vtmwlt59mcqb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98e7c2b5-126e-4d99-a480-38555e004a37
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/98e7c2b5-126e-4d99-a480-38555e004a37",
      "name": "98e7c2b5-126e-4d99-a480-38555e004a37",
      "status": "Failed",
      "startTime": "2026-07-23T15:32:15.307228267Z",
      "endTime": "2026-07-23T15:51:31.367394587Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc00115c360>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-hsklklhdjwx5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d324661b-7e24-4358-8d2f-caad268deb7d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d324661b-7e24-4358-8d2f-caad268deb7d",
      "name": "d324661b-7e24-4358-8d2f-caad268deb7d",
      "status": "Failed",
      "startTime": "2026-07-23T14:58:54.105750542Z",
      "endTime": "2026-07-23T15:18:00.560174863Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:104]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc00115c360>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-hsklklhdjwx5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d324661b-7e24-4358-8d2f-caad268deb7d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d324661b-7e24-4358-8d2f-caad268deb7d",
      "name": "d324661b-7e24-4358-8d2f-caad268deb7d",
      "status": "Failed",
      "startTime": "2026-07-23T14:58:54.105750542Z",
      "endTime": "2026-07-23T15:18:00.560174863Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
14:34:14 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:79]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000cec5c0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-hq8vhpn7hs7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13ec6fc7-0fe9-440e-b954-d90bbd276f01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13ec6fc7-0fe9-440e-b954-d90bbd276f01",
      "name": "13ec6fc7-0fe9-440e-b954-d90bbd276f01",
      "status": "Failed",
      "startTime": "2026-07-23T14:56:31.891603533Z",
      "endTime": "2026-07-23T15:15:50.558670399Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:79]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000cec5c0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-hq8vhpn7hs7z" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13ec6fc7-0fe9-440e-b954-d90bbd276f01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/13ec6fc7-0fe9-440e-b954-d90bbd276f01",
      "name": "13ec6fc7-0fe9-440e-b954-d90bbd276f01",
      "status": "Failed",
      "startTime": "2026-07-23T14:56:31.891603533Z",
      "endTime": "2026-07-23T15:15:50.558670399Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] <no_message>; [cosmosCluster] <no_message>; provider Microsoft.RedHatOpenShift
04:11:53 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:181]: failed to create HCP cluster "cp-ystream-upgrade-4-21-zxpbdh" with install version 4.20.30
Unexpected error:
    <*fmt.wrapError | 0xc000cf11c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-zxpbdh: failed waiting for cluster="cp-ystream-upgrade-4-21-zxpbdh" in resourcegroup="rg-cp-ystream-upgrade-4-21-4s5rv26vwmxd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73d26d21-c65b-4a19-9eb0-29e0ae5aacee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73d26d21-c65b-4a19-9eb0-29e0ae5aacee",
      "name": "73d26d21-c65b-4a19-9eb0-29e0ae5aacee",
      "status": "Failed",
      "startTime": "2026-07-23T05:03:31.056503296Z",
      "endTime": "2026-07-23T05:22:34.512235877Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:181]: failed to create HCP cluster "cp-ystream-upgrade-4-21-zxpbdh" with install version 4.20.30
Unexpected error:
    <*fmt.wrapError | 0xc000cf11c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-zxpbdh: failed waiting for cluster="cp-ystream-upgrade-4-21-zxpbdh" in resourcegroup="rg-cp-ystream-upgrade-4-21-4s5rv26vwmxd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73d26d21-c65b-4a19-9eb0-29e0ae5aacee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/73d26d21-c65b-4a19-9eb0-29e0ae5aacee",
      "name": "73d26d21-c65b-4a19-9eb0-29e0ae5aacee",
      "status": "Failed",
      "startTime": "2026-07-23T05:03:31.056503296Z",
      "endTime": "2026-07-23T05:22:34.512235877Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] \u003cno_message\u003e; [cosmosCluster] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a