CIHealth
<
Single Day: Oct 27▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: STG

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 23m 22s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 1
e2e (1)
VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
23:00:49 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-htnxwk" is viable
Unexpected error:
    <*errors.joinError | 0x28946bc08d20>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x28946b722320>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x28946b722300>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x28946b314d20>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x28946b60da40>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x28946b60da20>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x28946b518dc0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:168]: failed to verify HCP cluster "cluster-nightly-4-21-htnxwk" is viable
Unexpected error:
    <*errors.joinError | 0x28946bc08d20>: 
    VerifyBasicAccess failed: failed to list services: Unauthorized
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x28946b722320>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x28946b722300>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x28946b314d20>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x28946b60da40>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x28946b60da20>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x28946b518dc0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
2h 39m 53s branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel-all
post-goodmerged PR
No additional run metadata captured
 uksouth n/a passed n/a 0 n/a
2h 37m 58s branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel-all
post-goodmerged PR
No additional run metadata captured
 uksouth n/a passed n/a 0 n/a
2h 29m 43s branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel-all
post-goodmerged PR
No additional run metadata captured
 westus3 unknown failed n/a 12
unknown (12)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-b485rw" for cluster "cluster-candidate-4-23-b485rw"
Unexpected error:
    <*fmt.wrapError | 0x1075b7e74980>: 
    failed to create NodePool np-b485rw: failed waiting for nodepool="np-b485rw" for cluster "cluster-candidate-4-23-b485rw" in resourcegroup="rg-candidate-4-23-khfknr4zp64h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f65d5700-0d56-4f82-b528-c8a03c20b091
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f65d5700-0d56-4f82-b528-c8a03c20b091",
      "name": "f65d5700-0d56-4f82-b528-c8a03c20b091",
      "status": "Failed",
      "startTime": "2026-10-27T08:33:58.455860138Z",
      "endTime": "2026-10-27T08:52:58.299035711Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c0c5j0l8n8r8a3r-np-b485rw-67wmg-9gvgj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine c0c5j0l8n8r8a3r-np-b485rw-67wmg-bgdzh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c0c5j0l8n8r8a3r-np-b485rw-67wmg-9gvgj: Creating: virtualmachine creating or updating\nMachine c0c5j0l8n8r8a3r-np-b485rw-67wmg-bgdzh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f65d5700-0d56-4f82-b528-c8a03c20b091
            correlation ID: 9b1ecdea-62ca-4b99-996f-b8c59602813a
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-b485rw" for cluster "cluster-candidate-4-23-b485rw"
Unexpected error:
    <*fmt.wrapError | 0x1075b7e74980>: 
    failed to create NodePool np-b485rw: failed waiting for nodepool="np-b485rw" for cluster "cluster-candidate-4-23-b485rw" in resourcegroup="rg-candidate-4-23-khfknr4zp64h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f65d5700-0d56-4f82-b528-c8a03c20b091
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f65d5700-0d56-4f82-b528-c8a03c20b091",
      "name": "f65d5700-0d56-4f82-b528-c8a03c20b091",
      "status": "Failed",
      "startTime": "2026-10-27T08:33:58.455860138Z",
      "endTime": "2026-10-27T08:52:58.299035711Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c0c5j0l8n8r8a3r-np-b485rw-67wmg-9gvgj: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine c0c5j0l8n8r8a3r-np-b485rw-67wmg-bgdzh: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c0c5j0l8n8r8a3r-np-b485rw-67wmg-9gvgj: Creating: virtualmachine creating or updating\nMachine c0c5j0l8n8r8a3r-np-b485rw-67wmg-bgdzh: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f65d5700-0d56-4f82-b528-c8a03c20b091
            correlation ID: 9b1ecdea-62ca-4b99-996f-b8c59602813a
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-pwr5hm" for cluster "cluster-candidate-5-0-pwr5hm"
Unexpected error:
    <*fmt.wrapError | 0x1a6192d956a0>: 
    failed to create NodePool np-pwr5hm: failed waiting for nodepool="np-pwr5hm" for cluster "cluster-candidate-5-0-pwr5hm" in resourcegroup="rg-candidate-5-0-7hv6h2jrxcx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c22a5a79-f84e-4aa7-9d3b-91a7d7558e28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c22a5a79-f84e-4aa7-9d3b-91a7d7558e28",
      "name": "c22a5a79-f84e-4aa7-9d3b-91a7d7558e28",
      "status": "Failed",
      "startTime": "2026-10-27T08:36:44.781927018Z",
      "endTime": "2026-10-27T08:55:48.199837161Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h7y2l9q7w3r4t6i-np-pwr5hm-f6g86-k4b6d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h7y2l9q7w3r4t6i-np-pwr5hm-f6g86-trw27: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h7y2l9q7w3r4t6i-np-pwr5hm-f6g86-k4b6d: Creating: virtualmachine creating or updating\nMachine h7y2l9q7w3r4t6i-np-pwr5hm-f6g86-trw27: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c22a5a79-f84e-4aa7-9d3b-91a7d7558e28
            correlation ID: fc347c43-9125-439c-9002-0dcad1e9a2a5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-pwr5hm" for cluster "cluster-candidate-5-0-pwr5hm"
Unexpected error:
    <*fmt.wrapError | 0x1a6192d956a0>: 
    failed to create NodePool np-pwr5hm: failed waiting for nodepool="np-pwr5hm" for cluster "cluster-candidate-5-0-pwr5hm" in resourcegroup="rg-candidate-5-0-7hv6h2jrxcx9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c22a5a79-f84e-4aa7-9d3b-91a7d7558e28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c22a5a79-f84e-4aa7-9d3b-91a7d7558e28",
      "name": "c22a5a79-f84e-4aa7-9d3b-91a7d7558e28",
      "status": "Failed",
      "startTime": "2026-10-27T08:36:44.781927018Z",
      "endTime": "2026-10-27T08:55:48.199837161Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine h7y2l9q7w3r4t6i-np-pwr5hm-f6g86-k4b6d: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine h7y2l9q7w3r4t6i-np-pwr5hm-f6g86-trw27: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine h7y2l9q7w3r4t6i-np-pwr5hm-f6g86-k4b6d: Creating: virtualmachine creating or updating\nMachine h7y2l9q7w3r4t6i-np-pwr5hm-f6g86-trw27: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c22a5a79-f84e-4aa7-9d3b-91a7d7558e28
            correlation ID: fc347c43-9125-439c-9002-0dcad1e9a2a5
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-ml8dz8" for cluster "cluster-candidate-4-20-ml8dz8"
Unexpected error:
    <*fmt.wrapError | 0x25c240586440>: 
    failed to create NodePool np-ml8dz8: failed waiting for nodepool="np-ml8dz8" for cluster "cluster-candidate-4-20-ml8dz8" in resourcegroup="rg-candidate-4-20-jk94bqp4t7c4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca9d18db-ad83-4257-86ca-b626b3a1c04e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca9d18db-ad83-4257-86ca-b626b3a1c04e",
      "name": "ca9d18db-ad83-4257-86ca-b626b3a1c04e",
      "status": "Failed",
      "startTime": "2026-10-27T08:32:49.74394684Z",
      "endTime": "2026-10-27T08:51:58.323448782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t2v7o2w6e3c2h7k-np-ml8dz8-lm8lp-bw7gg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine t2v7o2w6e3c2h7k-np-ml8dz8-lm8lp-hkpd5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t2v7o2w6e3c2h7k-np-ml8dz8-lm8lp-bw7gg: Creating: virtualmachine creating or updating\nMachine t2v7o2w6e3c2h7k-np-ml8dz8-lm8lp-hkpd5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca9d18db-ad83-4257-86ca-b626b3a1c04e
            correlation ID: b6d3f936-bcc1-4c84-b01a-da9310e58e24
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-ml8dz8" for cluster "cluster-candidate-4-20-ml8dz8"
Unexpected error:
    <*fmt.wrapError | 0x25c240586440>: 
    failed to create NodePool np-ml8dz8: failed waiting for nodepool="np-ml8dz8" for cluster "cluster-candidate-4-20-ml8dz8" in resourcegroup="rg-candidate-4-20-jk94bqp4t7c4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca9d18db-ad83-4257-86ca-b626b3a1c04e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca9d18db-ad83-4257-86ca-b626b3a1c04e",
      "name": "ca9d18db-ad83-4257-86ca-b626b3a1c04e",
      "status": "Failed",
      "startTime": "2026-10-27T08:32:49.74394684Z",
      "endTime": "2026-10-27T08:51:58.323448782Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t2v7o2w6e3c2h7k-np-ml8dz8-lm8lp-bw7gg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine t2v7o2w6e3c2h7k-np-ml8dz8-lm8lp-hkpd5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t2v7o2w6e3c2h7k-np-ml8dz8-lm8lp-bw7gg: Creating: virtualmachine creating or updating\nMachine t2v7o2w6e3c2h7k-np-ml8dz8-lm8lp-hkpd5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ca9d18db-ad83-4257-86ca-b626b3a1c04e
            correlation ID: b6d3f936-bcc1-4c84-b01a-da9310e58e24
            client request ID: unavailable
occurred
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
07:33:22 UTC · failed at unknown · test Customer should be able to delete a nodepool · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3b12e2cba60>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4j5vtdb84h8v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
      "name": "083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
      "status": "Failed",
      "startTime": "2026-10-27T09:13:51.032005174Z",
      "endTime": "2026-10-27T09:32:53.54866754Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: Creating: virtualmachine creating or updating\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
            correlation ID: 01ff1c80-ae7b-4cd9-aeb7-68197786fff9
            client request ID: unavailable
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3b12e2cba60>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4j5vtdb84h8v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
      "name": "083c3fe4-0c43-4b6a-9905-e8c1aa97e192",
      "status": "Failed",
      "startTime": "2026-10-27T09:13:51.032005174Z",
      "endTime": "2026-10-27T09:32:53.54866754Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-nd4jp: Creating: virtualmachine creating or updating\nMachine o3t1v0i5b5o7c5l-np-delete-main-bg4kp-q624c: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/083c3fe4-0c43-4b6a-9905-e8c1aa97e192
            correlation ID: 01ff1c80-ae7b-4cd9-aeb7-68197786fff9
            client request ID: unavailable
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x3baee6990108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2gdwf4pbz22s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
      "name": "0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
      "status": "Failed",
      "startTime": "2026-10-27T08:12:10.541346261Z",
      "endTime": "2026-10-27T08:31:14.925265024Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
            correlation ID: 6e36e413-6dec-4393-bcb3-4788f0d108b3
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x3baee6990108>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-2gdwf4pbz22s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
      "name": "0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af",
      "status": "Failed",
      "startTime": "2026-10-27T08:12:10.541346261Z",
      "endTime": "2026-10-27T08:31:14.925265024Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-ffmd5-h94sn: Creating: virtualmachine creating or updating\nMachine cilium-cluster-cilium-np-ffmd5-hvcp5: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0e019fa7-e26d-44c3-b5c0-c1a3e7ecc4af
            correlation ID: 6e36e413-6dec-4393-bcb3-4788f0d108b3
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1401a4bfa220>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-76t2rcnvs776" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bcfb241d-404a-41b7-85ba-06ec88d78400
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bcfb241d-404a-41b7-85ba-06ec88d78400",
      "name": "bcfb241d-404a-41b7-85ba-06ec88d78400",
      "status": "Failed",
      "startTime": "2026-10-27T08:11:08.682187527Z",
      "endTime": "2026-10-27T08:30:15.063678012Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r6n8p2x9f8z6q7e-np-1-x9ghg-6ql2f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine r6n8p2x9f8z6q7e-np-1-x9ghg-q2s94: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r6n8p2x9f8z6q7e-np-1-x9ghg-6ql2f: Creating: virtualmachine creating or updating\nMachine r6n8p2x9f8z6q7e-np-1-x9ghg-q2s94: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bcfb241d-404a-41b7-85ba-06ec88d78400
            correlation ID: d86dd47b-4014-46c8-b8ef-73967dae5ad5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1401a4bfa220>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-76t2rcnvs776" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bcfb241d-404a-41b7-85ba-06ec88d78400
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bcfb241d-404a-41b7-85ba-06ec88d78400",
      "name": "bcfb241d-404a-41b7-85ba-06ec88d78400",
      "status": "Failed",
      "startTime": "2026-10-27T08:11:08.682187527Z",
      "endTime": "2026-10-27T08:30:15.063678012Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r6n8p2x9f8z6q7e-np-1-x9ghg-6ql2f: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine r6n8p2x9f8z6q7e-np-1-x9ghg-q2s94: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r6n8p2x9f8z6q7e-np-1-x9ghg-6ql2f: Creating: virtualmachine creating or updating\nMachine r6n8p2x9f8z6q7e-np-1-x9ghg-q2s94: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bcfb241d-404a-41b7-85ba-06ec88d78400
            correlation ID: d86dd47b-4014-46c8-b8ef-73967dae5ad5
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test Customer should be able to retrieve cluster and nodepool status active versions · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1520efb25aa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-g5mddw2znmwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191",
      "name": "a9c68b69-2f13-4771-88f2-779ee3c74191",
      "status": "Failed",
      "startTime": "2026-10-27T08:30:17.391270547Z",
      "endTime": "2026-10-27T08:49:18.18634688Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-gzw7k-gkpzc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v20261001-np-1-gzw7k-sxqb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-gzw7k-gkpzc: Creating: virtualmachine creating or updating\nMachine v20261001-np-1-gzw7k-sxqb6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
            correlation ID: c7660621-545a-4ad8-89c0-9dd90d63841c
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:96]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1520efb25aa0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-g5mddw2znmwk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191",
      "name": "a9c68b69-2f13-4771-88f2-779ee3c74191",
      "status": "Failed",
      "startTime": "2026-10-27T08:30:17.391270547Z",
      "endTime": "2026-10-27T08:49:18.18634688Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-gzw7k-gkpzc: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine v20261001-np-1-gzw7k-sxqb6: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-gzw7k-gkpzc: Creating: virtualmachine creating or updating\nMachine v20261001-np-1-gzw7k-sxqb6: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9c68b69-2f13-4771-88f2-779ee3c74191
            correlation ID: c7660621-545a-4ad8-89c0-9dd90d63841c
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-fpbjtx" for cluster "cluster-candidate-4-21-fpbjtx"
Unexpected error:
    <*fmt.wrapError | 0x324eadaf2a60>: 
    failed to create NodePool np-fpbjtx: failed waiting for nodepool="np-fpbjtx" for cluster "cluster-candidate-4-21-fpbjtx" in resourcegroup="rg-candidate-4-21-5f22mllfvqc4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a3967a25-cc4a-4063-9e34-8923a8b64863
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a3967a25-cc4a-4063-9e34-8923a8b64863",
      "name": "a3967a25-cc4a-4063-9e34-8923a8b64863",
      "status": "Failed",
      "startTime": "2026-10-27T08:33:18.660197992Z",
      "endTime": "2026-10-27T08:52:18.18082794Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o2k6z2j0v1z5d4j-np-fpbjtx-lg9g5-dzsl5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o2k6z2j0v1z5d4j-np-fpbjtx-lg9g5-sv4wl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o2k6z2j0v1z5d4j-np-fpbjtx-lg9g5-dzsl5: Creating: virtualmachine creating or updating\nMachine o2k6z2j0v1z5d4j-np-fpbjtx-lg9g5-sv4wl: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a3967a25-cc4a-4063-9e34-8923a8b64863
            correlation ID: 9548c536-1f3f-4571-b9bb-a8ccda502489
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-fpbjtx" for cluster "cluster-candidate-4-21-fpbjtx"
Unexpected error:
    <*fmt.wrapError | 0x324eadaf2a60>: 
    failed to create NodePool np-fpbjtx: failed waiting for nodepool="np-fpbjtx" for cluster "cluster-candidate-4-21-fpbjtx" in resourcegroup="rg-candidate-4-21-5f22mllfvqc4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a3967a25-cc4a-4063-9e34-8923a8b64863
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a3967a25-cc4a-4063-9e34-8923a8b64863",
      "name": "a3967a25-cc4a-4063-9e34-8923a8b64863",
      "status": "Failed",
      "startTime": "2026-10-27T08:33:18.660197992Z",
      "endTime": "2026-10-27T08:52:18.18082794Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o2k6z2j0v1z5d4j-np-fpbjtx-lg9g5-dzsl5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine o2k6z2j0v1z5d4j-np-fpbjtx-lg9g5-sv4wl: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o2k6z2j0v1z5d4j-np-fpbjtx-lg9g5-dzsl5: Creating: virtualmachine creating or updating\nMachine o2k6z2j0v1z5d4j-np-fpbjtx-lg9g5-sv4wl: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a3967a25-cc4a-4063-9e34-8923a8b64863
            correlation ID: 9548c536-1f3f-4571-b9bb-a8ccda502489
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-2pmcg4" for cluster "cluster-candidate-4-22-2pmcg4"
Unexpected error:
    <*fmt.wrapError | 0x144f200bc9a0>: 
    failed to create NodePool np-2pmcg4: failed waiting for nodepool="np-2pmcg4" for cluster "cluster-candidate-4-22-2pmcg4" in resourcegroup="rg-candidate-4-22-kwkd4shpnzkv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc208071-b38b-467a-86c5-b1a1983a3523
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc208071-b38b-467a-86c5-b1a1983a3523",
      "name": "cc208071-b38b-467a-86c5-b1a1983a3523",
      "status": "Failed",
      "startTime": "2026-10-27T08:34:38.028204768Z",
      "endTime": "2026-10-27T08:53:38.318204864Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j6w2a7x0u1f2u0h-np-2pmcg4-vfsxq-2v2b5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine j6w2a7x0u1f2u0h-np-2pmcg4-vfsxq-dmn7m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j6w2a7x0u1f2u0h-np-2pmcg4-vfsxq-2v2b5: Creating: virtualmachine creating or updating\nMachine j6w2a7x0u1f2u0h-np-2pmcg4-vfsxq-dmn7m: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc208071-b38b-467a-86c5-b1a1983a3523
            correlation ID: 27a80d85-0fd2-4584-9216-69874b20c19e
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-2pmcg4" for cluster "cluster-candidate-4-22-2pmcg4"
Unexpected error:
    <*fmt.wrapError | 0x144f200bc9a0>: 
    failed to create NodePool np-2pmcg4: failed waiting for nodepool="np-2pmcg4" for cluster "cluster-candidate-4-22-2pmcg4" in resourcegroup="rg-candidate-4-22-kwkd4shpnzkv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc208071-b38b-467a-86c5-b1a1983a3523
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc208071-b38b-467a-86c5-b1a1983a3523",
      "name": "cc208071-b38b-467a-86c5-b1a1983a3523",
      "status": "Failed",
      "startTime": "2026-10-27T08:34:38.028204768Z",
      "endTime": "2026-10-27T08:53:38.318204864Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j6w2a7x0u1f2u0h-np-2pmcg4-vfsxq-2v2b5: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine j6w2a7x0u1f2u0h-np-2pmcg4-vfsxq-dmn7m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j6w2a7x0u1f2u0h-np-2pmcg4-vfsxq-2v2b5: Creating: virtualmachine creating or updating\nMachine j6w2a7x0u1f2u0h-np-2pmcg4-vfsxq-dmn7m: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc208071-b38b-467a-86c5-b1a1983a3523
            correlation ID: 27a80d85-0fd2-4584-9216-69874b20c19e
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x21bec3e60060>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-p48wwqxwbhds" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a694363-a54b-4c78-9450-e4038850db77
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a694363-a54b-4c78-9450-e4038850db77",
      "name": "4a694363-a54b-4c78-9450-e4038850db77",
      "status": "Failed",
      "startTime": "2026-10-27T08:09:27.098324224Z",
      "endTime": "2026-10-27T08:28:35.105115624Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-n69xn-2rm9g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-n69xn-2rm9g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a694363-a54b-4c78-9450-e4038850db77
            correlation ID: ae9de2e2-5408-48a4-abcb-54d1d6de37d5
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x21bec3e60060>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-p48wwqxwbhds" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a694363-a54b-4c78-9450-e4038850db77
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a694363-a54b-4c78-9450-e4038850db77",
      "name": "4a694363-a54b-4c78-9450-e4038850db77",
      "status": "Failed",
      "startTime": "2026-10-27T08:09:27.098324224Z",
      "endTime": "2026-10-27T08:28:35.105115624Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-n69xn-2rm9g: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-n69xn-2rm9g: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4a694363-a54b-4c78-9450-e4038850db77
            correlation ID: ae9de2e2-5408-48a4-abcb-54d1d6de37d5
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m"
Unexpected error:
    <*fmt.wrapError | 0x2089be556480>: 
    failed to create NodePool np-jxwc9m: failed waiting for nodepool="np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m" in resourcegroup="rg-candidate-5-1-t5bnkh9ptx8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a",
      "name": "aa13a852-544e-4ae1-9ee9-db477af0802a",
      "status": "Failed",
      "startTime": "2026-10-27T08:36:20.928028726Z",
      "endTime": "2026-10-27T08:55:28.505111283Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: Creating: virtualmachine creating or updating\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
            correlation ID: 994d5b05-7775-48cd-93aa-f17d4c56d0ef
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:181]: failed to create node pool "np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m"
Unexpected error:
    <*fmt.wrapError | 0x2089be556480>: 
    failed to create NodePool np-jxwc9m: failed waiting for nodepool="np-jxwc9m" for cluster "cluster-candidate-5-1-jxwc9m" in resourcegroup="rg-candidate-5-1-t5bnkh9ptx8p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a",
      "name": "aa13a852-544e-4ae1-9ee9-db477af0802a",
      "status": "Failed",
      "startTime": "2026-10-27T08:36:20.928028726Z",
      "endTime": "2026-10-27T08:55:28.505111283Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-nmkn7: Creating: virtualmachine creating or updating\nMachine f5e7a9i9o6b7x5n-np-jxwc9m-l25lf-rxvtq: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa13a852-544e-4ae1-9ee9-db477af0802a
            correlation ID: 994d5b05-7775-48cd-93aa-f17d4c56d0ef
            client request ID: unavailable
occurred
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for AzureMachine to report spec.providerID; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: Creating: virtualmachine creating or updating; provider Microsoft.RedHatOpenShift
07:33:22 UTC · failed at unknown · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel/all
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xa0fb7d04138>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-xf9jw4l5cdpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58",
      "name": "0006ad13-87f3-4fc4-8b43-20306abddf58",
      "status": "Failed",
      "startTime": "2026-10-27T08:14:54.971524126Z",
      "endTime": "2026-10-27T08:33:58.162558969Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
            correlation ID: 7bfa7cc1-239c-4ae3-a9bc-578e94ecc5a3
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xa0fb7d04138>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-xf9jw4l5cdpf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58",
      "name": "0006ad13-87f3-4fc4-8b43-20306abddf58",
      "status": "Failed",
      "startTime": "2026-10-27T08:14:54.971524126Z",
      "endTime": "2026-10-27T08:33:58.162558969Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-7cl7p-8fh5m: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-7cl7p-fd22m: Creating: virtualmachine creating or updating\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0006ad13-87f3-4fc4-8b43-20306abddf58
            correlation ID: 7bfa7cc1-239c-4ae3-a9bc-578e94ecc5a3
            client request ID: unavailable
occurred
2h 53m 58s branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel-all
post-goodmerged PR
No additional run metadata captured
 uksouth n/a passed n/a 0 n/a
1h 54m 19s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 4
e2e (4)
VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0
02:00:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2nz4xc"
Expected success, but got an error:
    <*fmt.wrapError | 0x203abb848000>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0x203ab927a940>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:256]: default-ingress-cert must exist and be referenced by IngressController/default for cluster "cluster-candidate-5-0-2nz4xc"
Expected success, but got an error:
    <*fmt.wrapError | 0x203abb848000>: 
    VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
    {
        msg: "VerifyDefaultIngressServingCertificateSecretWiring timed out after 10m0.001s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        err: <*errors.errorString | 0x203ab927a940>{
            s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
        },
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
02:00:05 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x66df03e8b20>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x66df03e8b20>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
02:00:05 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.23 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x2de6fa3020f0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:528]: ACM must not create its ingress certificate secret
Expected success, but got an error:
    <*errors.errorString | 0x2de6fa3020f0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
02:00:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-wlxxtl"
Expected success, but got an error:
    <*errors.errorString | 0x30c9514302d0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:260]: ACM must not create its ingress certificate secret for cluster "cluster-candidate-4-23-wlxxtl"
Expected success, but got an error:
    <*errors.errorString | 0x30c9514302d0>: 
    ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed
    {
        s: "ACM ingress secret openshift-ingress/cluster-ingress-cert exists; expected ACM certificate delivery to be suppressed",
    }