CIHealth
<
Single Day: Jul 29▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)JobFailed atResultPRFailed testsDetails
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 2
e2e (2)
failed to create service principal: create service principal: error status code received from the API
23:00:40 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:127]: failed to create app registration with service principal
Unexpected error:
    <*fmt.wrapError | 0xc000e722c0>: 
    failed to create service principal: create service principal: error status code received from the API
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:127]: failed to create app registration with service principal
Unexpected error:
    <*fmt.wrapError | 0xc000e722c0>: 
    failed to create service principal: create service principal: error status code received from the API
    ...
occurred
failed to create service principal: create service principal: error status code received from the API
23:00:40 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:264]: failed to create app registration with service principal
Unexpected error:
    <*fmt.wrapError | 0xc000d25ea0>: 
    failed to create service principal: create service principal: error status code received from the API
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:264]: failed to create app registration with service principal
Unexpected error:
    <*fmt.wrapError | 0xc000d25ea0>: 
    failed to create service principal: create service principal: error status code received from the API
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 4
e2e (4)
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
18:28:59 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc00130a258>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-candidate-4-22-5pp8d59rjhfd", caused by: failed waiting for hcpCluster="cluster-candidate-4-22-qwv8sf" in resourcegroup="rg-candidate-4-22-5pp8d59rjhfd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-22-qwv8sf in resource group rg-candidate-4-22-5pp8d59rjhfd, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-candidate-4-22-qwv8sf" in resourcegroup="rg-candidate-4-22-5pp8d59rjhfd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-22-qwv8sf in resource group rg-candidate-4-22-5pp8d59rjhfd, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc00130a258>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-candidate-4-22-5pp8d59rjhfd", caused by: failed waiting for hcpCluster="cluster-candidate-4-22-qwv8sf" in resourcegroup="rg-candidate-4-22-5pp8d59rjhfd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-22-qwv8sf in resource group rg-candidate-4-22-5pp8d59rjhfd, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-candidate-4-22-qwv8sf" in resourcegroup="rg-candidate-4-22-5pp8d59rjhfd" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-candidate-4-22-qwv8sf in resource group rg-candidate-4-22-5pp8d59rjhfd, error: context deadline exceeded
    ...
occurred
add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
18:28:59 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:269]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc00291bce0>: 
    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:269]: failed to add password to app registration
Unexpected error:
    <*fmt.wrapErrors | 0xc00291bce0>: 
    add password after 1 attempts; last attempt error: error status code received from the API; polling error: error status code received from the API
    ...
occurred
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
18:28:59 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc000e48150>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="external-auth-cluster-hxfs6rqqk2l8", caused by: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-hxfs6rqqk2l8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster ea-cluster in resource group external-auth-cluster-hxfs6rqqk2l8, error: context deadline exceeded, error: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-hxfs6rqqk2l8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster ea-cluster in resource group external-auth-cluster-hxfs6rqqk2l8, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc000e48150>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="external-auth-cluster-hxfs6rqqk2l8", caused by: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-hxfs6rqqk2l8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster ea-cluster in resource group external-auth-cluster-hxfs6rqqk2l8, error: context deadline exceeded, error: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-hxfs6rqqk2l8" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster ea-cluster in resource group external-auth-cluster-hxfs6rqqk2l8, error: context deadline exceeded
    ...
occurred
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
18:28:59 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc001a2c0a8>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="ephemeral-osdisk-82hd5nnf8hsk", caused by: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-82hd5nnf8hsk" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster ephemeral-disk in resource group ephemeral-osdisk-82hd5nnf8hsk, error: context deadline exceeded, error: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-82hd5nnf8hsk" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster ephemeral-disk in resource group ephemeral-osdisk-82hd5nnf8hsk, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc001a2c0a8>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="ephemeral-osdisk-82hd5nnf8hsk", caused by: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-82hd5nnf8hsk" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster ephemeral-disk in resource group ephemeral-osdisk-82hd5nnf8hsk, error: context deadline exceeded, error: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-82hd5nnf8hsk" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster ephemeral-disk in resource group ephemeral-osdisk-82hd5nnf8hsk, error: context deadline exceeded
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [clusterServiceNodePoolStatus] <no_message>; provider Microsoft.RedHatOpenShift
11:20:50 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc002462078>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-gv7f9fvg24h7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1465f2b7-2fd3-4347-8208-672d33732fee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1465f2b7-2fd3-4347-8208-672d33732fee",
      "name": "1465f2b7-2fd3-4347-8208-672d33732fee",
      "status": "Failed",
      "startTime": "2026-07-29T11:50:37.856046587Z",
      "endTime": "2026-07-29T12:09:40.769017857Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:190]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc002462078>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-gv7f9fvg24h7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1465f2b7-2fd3-4347-8208-672d33732fee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1465f2b7-2fd3-4347-8208-672d33732fee",
      "name": "1465f2b7-2fd3-4347-8208-672d33732fee",
      "status": "Failed",
      "startTime": "2026-07-29T11:50:37.856046587Z",
      "endTime": "2026-07-29T12:09:40.769017857Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 8
e2e (8)
cluster entered terminal Failed state after role assignment deployment
04:59:16 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:219]: Timed out after 2520.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:215 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
04:59:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:79]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000e44d60>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-8t8h2tgjbrcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e7fe9e69-daaa-4497-9a2d-77e15fd09b7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e7fe9e69-daaa-4497-9a2d-77e15fd09b7c",
      "name": "e7fe9e69-daaa-4497-9a2d-77e15fd09b7c",
      "status": "Failed",
      "startTime": "2026-07-29T05:14:32.496403767Z",
      "endTime": "2026-07-29T05:33:46.625119368Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:79]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000e44d60>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-8t8h2tgjbrcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e7fe9e69-daaa-4497-9a2d-77e15fd09b7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e7fe9e69-daaa-4497-9a2d-77e15fd09b7c",
      "name": "e7fe9e69-daaa-4497-9a2d-77e15fd09b7c",
      "status": "Failed",
      "startTime": "2026-07-29T05:14:32.496403767Z",
      "endTime": "2026-07-29T05:33:46.625119368Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
04:59:16 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-20-7kl5nw54qr69/cluster-candidate-4-20-znj4vt should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a944e0>: 
    failed to create HCP cluster cluster-candidate-4-20-znj4vt: failed waiting for cluster="cluster-candidate-4-20-znj4vt" in resourcegroup="rg-candidate-4-20-7kl5nw54qr69" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/d9d17143-8160-42f2-ba83-7728bde83b14
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/d9d17143-8160-42f2-ba83-7728bde83b14",
      "name": "d9d17143-8160-42f2-ba83-7728bde83b14",
      "status": "Failed",
      "startTime": "2026-07-29T05:14:32.584383772Z",
      "endTime": "2026-07-29T05:33:46.687822425Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-20-7kl5nw54qr69/cluster-candidate-4-20-znj4vt should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a944e0>: 
    failed to create HCP cluster cluster-candidate-4-20-znj4vt: failed waiting for cluster="cluster-candidate-4-20-znj4vt" in resourcegroup="rg-candidate-4-20-7kl5nw54qr69" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/d9d17143-8160-42f2-ba83-7728bde83b14
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/d9d17143-8160-42f2-ba83-7728bde83b14",
      "name": "d9d17143-8160-42f2-ba83-7728bde83b14",
      "status": "Failed",
      "startTime": "2026-07-29T05:14:32.584383772Z",
      "endTime": "2026-07-29T05:33:46.687822425Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
04:59:16 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc00071afc0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/4772936f-ae74-485e-9e7e-40a7043cde4c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/4772936f-ae74-485e-9e7e-40a7043cde4c",
      "name": "4772936f-ae74-485e-9e7e-40a7043cde4c",
      "status": "Failed",
      "startTime": "2026-07-29T05:15:27.955090377Z",
      "endTime": "2026-07-29T05:34:46.282514393Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:138]: failed to wait for second cluster "basic-hcp-cluster2" to complete creation (timeout '20.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc00071afc0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/4772936f-ae74-485e-9e7e-40a7043cde4c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/4772936f-ae74-485e-9e7e-40a7043cde4c",
      "name": "4772936f-ae74-485e-9e7e-40a7043cde4c",
      "status": "Failed",
      "startTime": "2026-07-29T05:15:27.955090377Z",
      "endTime": "2026-07-29T05:34:46.282514393Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
04:59:16 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:159]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc001456540>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-ghbtv2frk8jv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/07ea6a1c-0e10-4d23-bee2-ed41df029a81
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/07ea6a1c-0e10-4d23-bee2-ed41df029a81",
      "name": "07ea6a1c-0e10-4d23-bee2-ed41df029a81",
      "status": "Failed",
      "startTime": "2026-07-29T05:14:31.670658764Z",
      "endTime": "2026-07-29T05:33:46.645534989Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:159]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc001456540>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-ghbtv2frk8jv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/07ea6a1c-0e10-4d23-bee2-ed41df029a81
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/07ea6a1c-0e10-4d23-bee2-ed41df029a81",
      "name": "07ea6a1c-0e10-4d23-bee2-ed41df029a81",
      "status": "Failed",
      "startTime": "2026-07-29T05:14:31.670658764Z",
      "endTime": "2026-07-29T05:33:46.645534989Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
04:59:16 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:87]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000ef86c0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-krvt2jsl464l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e72a9dba-3b99-45a7-a7d5-09149827f4b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e72a9dba-3b99-45a7-a7d5-09149827f4b7",
      "name": "e72a9dba-3b99-45a7-a7d5-09149827f4b7",
      "status": "Failed",
      "startTime": "2026-07-29T05:14:32.010755493Z",
      "endTime": "2026-07-29T05:33:46.238454371Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:87]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000ef86c0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-krvt2jsl464l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e72a9dba-3b99-45a7-a7d5-09149827f4b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e72a9dba-3b99-45a7-a7d5-09149827f4b7",
      "name": "e72a9dba-3b99-45a7-a7d5-09149827f4b7",
      "status": "Failed",
      "startTime": "2026-07-29T05:14:32.010755493Z",
      "endTime": "2026-07-29T05:33:46.238454371Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
04:59:16 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-21-p4kjqctgmzk6/cluster-candidate-4-21-glndl7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000644800>: 
    failed to create HCP cluster cluster-candidate-4-21-glndl7: failed waiting for cluster="cluster-candidate-4-21-glndl7" in resourcegroup="rg-candidate-4-21-p4kjqctgmzk6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e586cdc4-e3f0-4000-a84e-66355c4b12ff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e586cdc4-e3f0-4000-a84e-66355c4b12ff",
      "name": "e586cdc4-e3f0-4000-a84e-66355c4b12ff",
      "status": "Failed",
      "startTime": "2026-07-29T05:17:47.986658491Z",
      "endTime": "2026-07-29T05:37:06.506828647Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-21-p4kjqctgmzk6/cluster-candidate-4-21-glndl7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000644800>: 
    failed to create HCP cluster cluster-candidate-4-21-glndl7: failed waiting for cluster="cluster-candidate-4-21-glndl7" in resourcegroup="rg-candidate-4-21-p4kjqctgmzk6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e586cdc4-e3f0-4000-a84e-66355c4b12ff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/e586cdc4-e3f0-4000-a84e-66355c4b12ff",
      "name": "e586cdc4-e3f0-4000-a84e-66355c4b12ff",
      "status": "Failed",
      "startTime": "2026-07-29T05:17:47.986658491Z",
      "endTime": "2026-07-29T05:37:06.506828647Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
04:59:16 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0006a2180>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jp5gqq6r46ph" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f30d0249-fe7e-4dbb-ab0e-f68c73be2bba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f30d0249-fe7e-4dbb-ab0e-f68c73be2bba",
      "name": "f30d0249-fe7e-4dbb-ab0e-f68c73be2bba",
      "status": "Failed",
      "startTime": "2026-07-29T05:14:31.662472487Z",
      "endTime": "2026-07-29T05:33:46.618172895Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0006a2180>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jp5gqq6r46ph" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f30d0249-fe7e-4dbb-ab0e-f68c73be2bba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/f30d0249-fe7e-4dbb-ab0e-f68c73be2bba",
      "name": "f30d0249-fe7e-4dbb-ab0e-f68c73be2bba",
      "status": "Failed",
      "startTime": "2026-07-29T05:14:31.662472487Z",
      "endTime": "2026-07-29T05:33:46.618172895Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a passed n/a 0 n/a
periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail message [clusterServiceClusterStatus] <no_message>; provider Microsoft.RedHatOpenShift
02:00:39 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-21-fvwltc4gz4f6/cluster-candidate-4-21-49lzb5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b2d780>: 
    failed to create HCP cluster cluster-candidate-4-21-49lzb5: failed waiting for cluster="cluster-candidate-4-21-49lzb5" in resourcegroup="rg-candidate-4-21-fvwltc4gz4f6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d29f20f-5fbd-42e5-87c0-2a58dcea72d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d29f20f-5fbd-42e5-87c0-2a58dcea72d4",
      "name": "1d29f20f-5fbd-42e5-87c0-2a58dcea72d4",
      "status": "Failed",
      "startTime": "2026-07-29T02:06:05.891858654Z",
      "endTime": "2026-07-29T02:25:10.15756699Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:120]: HCP cluster rg-candidate-4-21-fvwltc4gz4f6/cluster-candidate-4-21-49lzb5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b2d780>: 
    failed to create HCP cluster cluster-candidate-4-21-49lzb5: failed waiting for cluster="cluster-candidate-4-21-49lzb5" in resourcegroup="rg-candidate-4-21-fvwltc4gz4f6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d29f20f-5fbd-42e5-87c0-2a58dcea72d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1d29f20f-5fbd-42e5-87c0-2a58dcea72d4",
      "name": "1d29f20f-5fbd-42e5-87c0-2a58dcea72d4",
      "status": "Failed",
      "startTime": "2026-07-29T02:06:05.891858654Z",
      "endTime": "2026-07-29T02:25:10.15756699Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceClusterStatus] \u003cno_message\u003e"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 e2e failed n/a 1
e2e (1)
DeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded
00:17:13 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc0016ec330>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="idms-jn69xnhg7z62", caused by: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jn69xnhg7z62" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-jn69xnhg7z62, error: context deadline exceeded, error: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jn69xnhg7z62" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-jn69xnhg7z62, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:285]: Unexpected error:
    <*errors.joinError | 0xc0016ec330>: 
    failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="idms-jn69xnhg7z62", caused by: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jn69xnhg7z62" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-jn69xnhg7z62, error: context deadline exceeded, error: failed waiting for hcpCluster="idms-e2e-hcp-cluster" in resourcegroup="idms-jn69xnhg7z62" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster idms-e2e-hcp-cluster in resource group idms-jn69xnhg7z62, error: context deadline exceeded
    ...
occurred