CIHealth
<
Single Day: Oct 1▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: STG

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
1h 2m 0s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
2h 28m 58s branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 13
e2e (13)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
22:19:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd"
Unexpected error:
    <*fmt.wrapError | 0xc001012e40>: 
    failed to create NodePool np-6g6tzd: failed waiting for nodepool="np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd" in resourcegroup="rg-candidate-4-22-xz7x2977wmz2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "name": "3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:22.461339415Z",
      "endTime": "2026-10-01T23:41:27.247269633Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: Creating: virtualmachine creating or updating\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd"
Unexpected error:
    <*fmt.wrapError | 0xc001012e40>: 
    failed to create NodePool np-6g6tzd: failed waiting for nodepool="np-6g6tzd" for cluster "cluster-candidate-4-22-6g6tzd" in resourcegroup="rg-candidate-4-22-xz7x2977wmz2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "name": "3c484f73-f02b-4ece-b374-6bbafdd0e00e",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:22.461339415Z",
      "endTime": "2026-10-01T23:41:27.247269633Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-564bh: Creating: virtualmachine creating or updating\nMachine m6u9b8d4j0a9a8z-np-6g6tzd-g5w62-777td: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
22:19:29 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000f50a20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-6svllmjvpbpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b",
      "name": "7c5e553a-2bca-431a-819c-94122ad78b6b",
      "status": "Failed",
      "startTime": "2026-10-02T00:04:50.703195637Z",
      "endTime": "2026-10-02T00:23:53.181881109Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
    <*fmt.wrapError | 0xc000f50a20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-6svllmjvpbpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7c5e553a-2bca-431a-819c-94122ad78b6b",
      "name": "7c5e553a-2bca-431a-819c-94122ad78b6b",
      "status": "Failed",
      "startTime": "2026-10-02T00:04:50.703195637Z",
      "endTime": "2026-10-02T00:23:53.181881109Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-6svllmjvpbpf--managed/providers/Microsoft.Compute/virtualMachines/i6r6j5b5y3t4r7g-np-1-tzs6p-6v7dr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
22:19:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-58mkpz" for cluster "cluster-candidate-4-21-58mkpz"
Unexpected error:
    <*fmt.wrapError | 0xc000cee380>: 
    failed to create NodePool np-58mkpz: failed waiting for nodepool="np-58mkpz" for cluster "cluster-candidate-4-21-58mkpz" in resourcegroup="rg-candidate-4-21-8w4nf2bg8hwt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/22afbbb9-6f81-4f16-a70d-aa16064ca2b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/22afbbb9-6f81-4f16-a70d-aa16064ca2b5",
      "name": "22afbbb9-6f81-4f16-a70d-aa16064ca2b5",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:00.007708534Z",
      "endTime": "2026-10-01T23:41:07.215796872Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-w8q2v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-xfhks: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-w8q2v: Creating: virtualmachine creating or updating\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-xfhks: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-58mkpz" for cluster "cluster-candidate-4-21-58mkpz"
Unexpected error:
    <*fmt.wrapError | 0xc000cee380>: 
    failed to create NodePool np-58mkpz: failed waiting for nodepool="np-58mkpz" for cluster "cluster-candidate-4-21-58mkpz" in resourcegroup="rg-candidate-4-21-8w4nf2bg8hwt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/22afbbb9-6f81-4f16-a70d-aa16064ca2b5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/22afbbb9-6f81-4f16-a70d-aa16064ca2b5",
      "name": "22afbbb9-6f81-4f16-a70d-aa16064ca2b5",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:00.007708534Z",
      "endTime": "2026-10-01T23:41:07.215796872Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-w8q2v: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-xfhks: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-w8q2v: Creating: virtualmachine creating or updating\nMachine k3l1b7w5a7b5w3u-np-58mkpz-kgdz9-xfhks: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
22:19:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nz54xv" for cluster "cluster-candidate-4-23-nz54xv"
Unexpected error:
    <*fmt.wrapError | 0xc000dbc340>: 
    failed to create NodePool np-nz54xv: failed waiting for nodepool="np-nz54xv" for cluster "cluster-candidate-4-23-nz54xv" in resourcegroup="rg-candidate-4-23-87g9stccjxzb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5",
      "name": "7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:46.805553224Z",
      "endTime": "2026-10-01T23:41:47.237736739Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-q48tw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-qpx5x: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-q48tw: Creating: virtualmachine creating or updating\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-qpx5x: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nz54xv" for cluster "cluster-candidate-4-23-nz54xv"
Unexpected error:
    <*fmt.wrapError | 0xc000dbc340>: 
    failed to create NodePool np-nz54xv: failed waiting for nodepool="np-nz54xv" for cluster "cluster-candidate-4-23-nz54xv" in resourcegroup="rg-candidate-4-23-87g9stccjxzb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5",
      "name": "7507c59e-c9ca-4318-bcfd-f4b2fc2e72d5",
      "status": "Failed",
      "startTime": "2026-10-01T23:22:46.805553224Z",
      "endTime": "2026-10-01T23:41:47.237736739Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-q48tw: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-qpx5x: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-q48tw: Creating: virtualmachine creating or updating\nMachine d9g4u8w0a7x2b4h-np-nz54xv-qb8xb-qpx5x: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
22:19:29 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000a47540>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-rj7chpbn7z2c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dd4c07c5-fe2d-47bf-9ef6-ea2171664623
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dd4c07c5-fe2d-47bf-9ef6-ea2171664623",
      "name": "dd4c07c5-fe2d-47bf-9ef6-ea2171664623",
      "status": "Failed",
      "startTime": "2026-10-01T22:58:07.666255851Z",
      "endTime": "2026-10-01T23:17:14.973053483Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-9d89g-jglhd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-9d89g-jglhd: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0xc000a47540>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-rj7chpbn7z2c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dd4c07c5-fe2d-47bf-9ef6-ea2171664623
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/dd4c07c5-fe2d-47bf-9ef6-ea2171664623",
      "name": "dd4c07c5-fe2d-47bf-9ef6-ea2171664623",
      "status": "Failed",
      "startTime": "2026-10-01T22:58:07.666255851Z",
      "endTime": "2026-10-01T23:17:14.973053483Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-9d89g-jglhd: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-9d89g-jglhd: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
22:19:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8gxplz" for cluster "cluster-candidate-5-1-8gxplz"
Unexpected error:
    <*fmt.wrapError | 0xc000acaa40>: 
    failed to create NodePool np-8gxplz: failed waiting for nodepool="np-8gxplz" for cluster "cluster-candidate-5-1-8gxplz" in resourcegroup="rg-candidate-5-1-kvc5d4thszfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bde76db6-b7ac-422e-8660-ba8904affbd3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bde76db6-b7ac-422e-8660-ba8904affbd3",
      "name": "bde76db6-b7ac-422e-8660-ba8904affbd3",
      "status": "Failed",
      "startTime": "2026-10-01T23:26:31.681894722Z",
      "endTime": "2026-10-01T23:45:37.242801753Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-245qj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-2577p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-245qj: Creating: virtualmachine creating or updating\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-2577p: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8gxplz" for cluster "cluster-candidate-5-1-8gxplz"
Unexpected error:
    <*fmt.wrapError | 0xc000acaa40>: 
    failed to create NodePool np-8gxplz: failed waiting for nodepool="np-8gxplz" for cluster "cluster-candidate-5-1-8gxplz" in resourcegroup="rg-candidate-5-1-kvc5d4thszfq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bde76db6-b7ac-422e-8660-ba8904affbd3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bde76db6-b7ac-422e-8660-ba8904affbd3",
      "name": "bde76db6-b7ac-422e-8660-ba8904affbd3",
      "status": "Failed",
      "startTime": "2026-10-01T23:26:31.681894722Z",
      "endTime": "2026-10-01T23:45:37.242801753Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-245qj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-2577p: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-245qj: Creating: virtualmachine creating or updating\nMachine o5t7r1t3k4e4s3h-np-8gxplz-bxndq-2577p: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
22:19:29 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019901c8>: 
    resource group "oidc-wi-nq9xg56z8ptc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-nq9xg56z8ptc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "name": "b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "status": "Failed",
      "startTime": "2026-10-02T00:11:10.101784102Z",
      "endTime": "2026-10-02T00:35:13.452817702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t6a9r7piu3mrhnemk9adlcvlla5mb7v still exists (deletion dispatched at 2026-10-02T00:11:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019901c8>: 
    resource group "oidc-wi-nq9xg56z8ptc": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="oidc-wi-cluster" in resourcegroup="oidc-wi-nq9xg56z8ptc" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "name": "b2411afb-26dd-4be7-a31a-e99647e9b52e",
      "status": "Failed",
      "startTime": "2026-10-02T00:11:10.101784102Z",
      "endTime": "2026-10-02T00:35:13.452817702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t6a9r7piu3mrhnemk9adlcvlla5mb7v still exists (deletion dispatched at 2026-10-02T00:11:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
22:19:29 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000d82400>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-g79pk8vm4jx6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "name": "d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "status": "Failed",
      "startTime": "2026-10-01T23:53:42.212331844Z",
      "endTime": "2026-10-02T00:12:43.153935456Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: Creating: virtualmachine creating or updating\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc000d82400>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-g79pk8vm4jx6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "name": "d7afe39e-2daf-4d6c-a78a-cef81fb1f217",
      "status": "Failed",
      "startTime": "2026-10-01T23:53:42.212331844Z",
      "endTime": "2026-10-02T00:12:43.153935456Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-c6ctv: Creating: virtualmachine creating or updating\nMachine i1n3h3r5l8p6b0o-np-delete-main-qzd58-qt8wr: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
22:19:29 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001f200c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4rdzlcbp8jqd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931",
      "name": "7941e9fc-eabb-4912-b47f-1619c832d931",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:15.574287096Z",
      "endTime": "2026-10-01T23:21:15.149755975Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc001f200c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4rdzlcbp8jqd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7941e9fc-eabb-4912-b47f-1619c832d931",
      "name": "7941e9fc-eabb-4912-b47f-1619c832d931",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:15.574287096Z",
      "endTime": "2026-10-01T23:21:15.149755975Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-tsk7z-g2d4q: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-tsk7z-rcxdh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
22:19:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-k96n75" for cluster "cluster-candidate-5-0-k96n75"
Unexpected error:
    <*fmt.wrapError | 0xc000a16440>: 
    failed to create NodePool np-k96n75: failed waiting for nodepool="np-k96n75" for cluster "cluster-candidate-5-0-k96n75" in resourcegroup="rg-candidate-5-0-gclccxznszk7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "name": "e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "status": "Failed",
      "startTime": "2026-10-01T23:23:36.842890739Z",
      "endTime": "2026-10-01T23:42:37.244899482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: Creating: virtualmachine creating or updating\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-k96n75" for cluster "cluster-candidate-5-0-k96n75"
Unexpected error:
    <*fmt.wrapError | 0xc000a16440>: 
    failed to create NodePool np-k96n75: failed waiting for nodepool="np-k96n75" for cluster "cluster-candidate-5-0-k96n75" in resourcegroup="rg-candidate-5-0-gclccxznszk7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "name": "e0f4877f-a7bc-408a-891f-bd985b38bfd8",
      "status": "Failed",
      "startTime": "2026-10-01T23:23:36.842890739Z",
      "endTime": "2026-10-01T23:42:37.244899482Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-6t7jp: Creating: virtualmachine creating or updating\nMachine l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-gclccxznszk7-candidate-k96n75--managed/providers/Microsoft.Compute/virtualMachines/l0o8x0e0c2l1i0q-np-k96n75-6284m-gt7km to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
22:19:29 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001c16198>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-mrv84pkzs8gp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "name": "c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:38.499943334Z",
      "endTime": "2026-10-01T23:21:45.00235976Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc001c16198>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-mrv84pkzs8gp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "name": "c363e8fc-7ec7-47fd-882a-c1fa371683da",
      "status": "Failed",
      "startTime": "2026-10-01T23:02:38.499943334Z",
      "endTime": "2026-10-01T23:21:45.00235976Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-rjv4k-5xzgh: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-rjv4k-7mcss: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
22:19:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bl9ftv" for cluster "cluster-candidate-4-20-bl9ftv"
Unexpected error:
    <*fmt.wrapError | 0xc001076a40>: 
    failed to create NodePool np-bl9ftv: failed waiting for nodepool="np-bl9ftv" for cluster "cluster-candidate-4-20-bl9ftv" in resourcegroup="rg-candidate-4-20-x4gs2n6gstps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c",
      "name": "c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c",
      "status": "Failed",
      "startTime": "2026-10-01T23:21:31.953266163Z",
      "endTime": "2026-10-01T23:40:37.297291005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-4tjkc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-pjccm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-4tjkc: Creating: virtualmachine creating or updating\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-pjccm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-bl9ftv" for cluster "cluster-candidate-4-20-bl9ftv"
Unexpected error:
    <*fmt.wrapError | 0xc001076a40>: 
    failed to create NodePool np-bl9ftv: failed waiting for nodepool="np-bl9ftv" for cluster "cluster-candidate-4-20-bl9ftv" in resourcegroup="rg-candidate-4-20-x4gs2n6gstps" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c",
      "name": "c10fd3d9-c2fc-4d77-bb0e-8b5b7e8cde2c",
      "status": "Failed",
      "startTime": "2026-10-01T23:21:31.953266163Z",
      "endTime": "2026-10-01T23:40:37.297291005Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-4tjkc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-pjccm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-4tjkc: Creating: virtualmachine creating or updating\nMachine i3l0t0t0y7o2p4y-np-bl9ftv-t8xwv-pjccm: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
22:19:29 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0009252c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-tfpfzw4vzd4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604",
      "name": "8e650b8c-c903-48db-ae29-20e1c0f74604",
      "status": "Failed",
      "startTime": "2026-10-01T23:01:41.16595227Z",
      "endTime": "2026-10-01T23:20:45.151714689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: Creating: virtualmachine creating or updating\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0009252c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-tfpfzw4vzd4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8e650b8c-c903-48db-ae29-20e1c0f74604",
      "name": "8e650b8c-c903-48db-ae29-20e1c0f74604",
      "status": "Failed",
      "startTime": "2026-10-01T23:01:41.16595227Z",
      "endTime": "2026-10-01T23:20:45.151714689Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-8nqmx: Creating: virtualmachine creating or updating\nMachine w8n1y8f9d9k3r1o-np-1-mllsn-dpzt8: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 33m 37s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 13
e2e (13)
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
02:00:28 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-c4cdqr" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00120fcb0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-bpfkv76t7qfk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-c4cdqr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-c4cdqr" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xc00120fcb0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-bpfkv76t7qfk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-c4cdqr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
02:00:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-kwx4gf" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0014a3d10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-l8ml659lgnj5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-kwx4gf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-5-0-kwx4gf" to follow minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xc0014a3d10>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-l8ml659lgnj5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-kwx4gf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
02:00:28 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-bngfn7" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000cf5770>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-q9xpw68pgwb7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-bngfn7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-bngfn7" to 4.21
Unexpected error:
    <*exported.ResponseError | 0xc000cf5770>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-q9xpw68pgwb7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-bngfn7
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
02:00:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-222nr2" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000993530>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-vf9snh5jjb8g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-222nr2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-20-222nr2" to follow minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0xc000993530>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-vf9snh5jjb8g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-222nr2
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:28 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr"
Unexpected error:
    <*fmt.wrapError | 0xc00081f300>: 
    failed to create NodePool np-8l66wr: failed waiting for nodepool="np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr" in resourcegroup="rg-candidate-4-20-66r4zk7wm6rk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404",
      "name": "1513980f-1a90-4c5b-865f-c0a3b6601404",
      "status": "Failed",
      "startTime": "2026-10-01T03:20:14.931117191Z",
      "endTime": "2026-10-01T03:39:22.388104494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr"
Unexpected error:
    <*fmt.wrapError | 0xc00081f300>: 
    failed to create NodePool np-8l66wr: failed waiting for nodepool="np-8l66wr" for cluster "cluster-candidate-4-20-8l66wr" in resourcegroup="rg-candidate-4-20-66r4zk7wm6rk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1513980f-1a90-4c5b-865f-c0a3b6601404",
      "name": "1513980f-1a90-4c5b-865f-c0a3b6601404",
      "status": "Failed",
      "startTime": "2026-10-01T03:20:14.931117191Z",
      "endTime": "2026-10-01T03:39:22.388104494Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-66r4zk7wm6rk-candidate-8l66wr--managed/providers/Microsoft.Compute/virtualMachines/y7w9c5q7k5c5l6z-np-8l66wr-wzjtv-cnmkc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
02:00:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-sq7dhp" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc0015c4e70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-vjpbzl2jbr62/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-sq7dhp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-23-sq7dhp" to follow minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0xc0015c4e70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-vjpbzl2jbr62/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-sq7dhp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:28 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc00171ed20>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-kc5k44mb955m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdbbe915-46e7-47c2-be0d-e2cbc376488e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdbbe915-46e7-47c2-be0d-e2cbc376488e",
      "name": "bdbbe915-46e7-47c2-be0d-e2cbc376488e",
      "status": "Failed",
      "startTime": "2026-10-01T02:58:20.214032414Z",
      "endTime": "2026-10-01T03:17:20.370385488Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-wwgqk-cz84n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kc5k44mb955m--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-wwgqk-cz84n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-wwgqk-cz84n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kc5k44mb955m--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-wwgqk-cz84n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc00171ed20>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-kc5k44mb955m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdbbe915-46e7-47c2-be0d-e2cbc376488e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bdbbe915-46e7-47c2-be0d-e2cbc376488e",
      "name": "bdbbe915-46e7-47c2-be0d-e2cbc376488e",
      "status": "Failed",
      "startTime": "2026-10-01T02:58:20.214032414Z",
      "endTime": "2026-10-01T03:17:20.370385488Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-wwgqk-cz84n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kc5k44mb955m--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-wwgqk-cz84n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine cilium-cl-cilium-np-wwgqk-cz84n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kc5k44mb955m--managed/providers/Microsoft.Compute/virtualMachines/cilium-cl-cilium-np-wwgqk-cz84n to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
02:00:28 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-hwc6dp" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00164d620>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-txlxhwglc5dq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-hwc6dp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-hwc6dp" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xc00164d620>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-txlxhwglc5dq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-hwc6dp
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
02:00:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-x5rqn9" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00163b1d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-d779fckrdm2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-x5rqn9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-22-x5rqn9" to follow minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0xc00163b1d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-d779fckrdm2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-x5rqn9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
02:00:28 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-kbwzzs" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc00104fe60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qb6v87bszxhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-kbwzzs
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-kbwzzs" to 5.1
Unexpected error:
    <*exported.ResponseError | 0xc00104fe60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-qb6v87bszxhb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-kbwzzs
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
02:00:28 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-psr6l5" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001800d20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-rn244qh65ldr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-psr6l5
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-psr6l5" to 4.22
Unexpected error:
    <*exported.ResponseError | 0xc001800d20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-rn244qh65ldr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-psr6l5
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidNullTagValue; detail message Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null.; provider Microsoft.RedHatOpenShift
02:00:28 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-pr7dlf" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0013e6d20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4ds2lzg2vct6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-pr7dlf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:150]: failed to remove the exact-version pin from cluster "cluster-zstream-4-21-pr7dlf" to follow minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xc0013e6d20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-4ds2lzg2vct6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-pr7dlf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidNullTagValue
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidNullTagValue",
        "message": "Invalid tag value. The following tags 'aro-hcp.experimental.cluster.control-plane-exact-version' have a null value. Tag value cannot be null."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:28 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000d6ac40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-b7k2xmx45mmg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f67a4120-58f4-48a2-b9dd-9ee7da78a289
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f67a4120-58f4-48a2-b9dd-9ee7da78a289",
      "name": "f67a4120-58f4-48a2-b9dd-9ee7da78a289",
      "status": "Failed",
      "startTime": "2026-10-01T02:57:30.013121006Z",
      "endTime": "2026-10-01T03:16:30.182282415Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h5u8t2s6s2i7w5q-np-1-78d96-vn4sd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-b7k2xmx45mmg--managed/providers/Microsoft.Compute/virtualMachines/h5u8t2s6s2i7w5q-np-1-78d96-vn4sd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h5u8t2s6s2i7w5q-np-1-78d96-vn4sd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-b7k2xmx45mmg--managed/providers/Microsoft.Compute/virtualMachines/h5u8t2s6s2i7w5q-np-1-78d96-vn4sd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000d6ac40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-b7k2xmx45mmg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f67a4120-58f4-48a2-b9dd-9ee7da78a289
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f67a4120-58f4-48a2-b9dd-9ee7da78a289",
      "name": "f67a4120-58f4-48a2-b9dd-9ee7da78a289",
      "status": "Failed",
      "startTime": "2026-10-01T02:57:30.013121006Z",
      "endTime": "2026-10-01T03:16:30.182282415Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine h5u8t2s6s2i7w5q-np-1-78d96-vn4sd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-b7k2xmx45mmg--managed/providers/Microsoft.Compute/virtualMachines/h5u8t2s6s2i7w5q-np-1-78d96-vn4sd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine h5u8t2s6s2i7w5q-np-1-78d96-vn4sd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-b7k2xmx45mmg--managed/providers/Microsoft.Compute/virtualMachines/h5u8t2s6s2i7w5q-np-1-78d96-vn4sd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 33m 47s branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 1
e2e (1)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
00:10:48 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000884ae0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-gcjj2967bz4p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d64d0e1-6be5-43b3-88bc-d71f31dca2c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d64d0e1-6be5-43b3-88bc-d71f31dca2c3",
      "name": "8d64d0e1-6be5-43b3-88bc-d71f31dca2c3",
      "status": "Failed",
      "startTime": "2026-10-01T01:34:57.093691985Z",
      "endTime": "2026-10-01T01:54:01.709819246Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gcjj2967bz4p--managed/providers/Microsoft.Compute/virtualMachines/z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gcjj2967bz4p--managed/providers/Microsoft.Compute/virtualMachines/z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000884ae0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-gcjj2967bz4p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d64d0e1-6be5-43b3-88bc-d71f31dca2c3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8d64d0e1-6be5-43b3-88bc-d71f31dca2c3",
      "name": "8d64d0e1-6be5-43b3-88bc-d71f31dca2c3",
      "status": "Failed",
      "startTime": "2026-10-01T01:34:57.093691985Z",
      "endTime": "2026-10-01T01:54:01.709819246Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gcjj2967bz4p--managed/providers/Microsoft.Compute/virtualMachines/z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-gcjj2967bz4p--managed/providers/Microsoft.Compute/virtualMachines/z9s5e8x2m9t9g2t-autoscale-noaz-88jfb-tfvcz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred