CIHealth
<
Single Day: Oct 8▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: STG

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
1h 32m 36s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 6
e2e (6)
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x3c35adc80030>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-clb6t6k9tfdf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x3c35adc80030>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-clb6t6k9tfdf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
VerifyBasicAccess failed: failed to list services: Unauthorized
23:00:38 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-20-g5bgdz" is viable
Unexpected error:
    <*errors.joinError | 0x2608a89861e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2608a90941e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2608a90941c0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2608a81192c0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2608a86f4180>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2608a86f4100>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2608a7f286e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:135]: failed to verify HCP cluster "cluster-nightly-4-20-g5bgdz" is viable
Unexpected error:
    <*errors.joinError | 0x2608a89861e0>: 
    VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized
    VerifyBasicAccess failed: failed to list services: Unauthorized
    {
        errs: [
            <*fmt.wrapError | 0x2608a90941e0>{
                msg: "VerifyAllAPIServicesAvailable failed: failed to list all APIServices: Unauthorized",
                err: <*fmt.wrapError | 0x2608a90941c0>{
                    msg: "failed to list all APIServices: Unauthorized",
                    err: <*errors.StatusError | 0x2608a81192c0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
            <*fmt.wrapError | 0x2608a86f4180>{
                msg: "VerifyBasicAccess failed: failed to list services: Unauthorized",
                err: <*fmt.wrapError | 0x2608a86f4100>{
                    msg: "failed to list services: Unauthorized",
                    err: <*errors.StatusError | 0x2608a7f286e0>{
                        ErrStatus: {
                            TypeMeta: {Kind: "", APIVersion: ""},
                            ListMeta: {
                                SelfLink: "",
                                ResourceVersion: "",
                                Continue: "",
                                RemainingItemCount: nil,
                                ShardInfo: nil,
                            },
                            Status: "Failure",
                            Message: "Unauthorized",
                            Reason: "Unauthorized",
                            Details: nil,
                            Code: 401,
                        },
                    },
                },
            },
        ],
    }
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:38 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x226dedcde000>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-c4d5nkzpw5nz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x226dedcde000>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-c4d5nkzpw5nz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:38 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x18f7dc4001e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-v2s5z7rdtt9b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x18f7dc4001e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-v2s5z7rdtt9b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:38 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x2add6a662db0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-wkvrqn6clxhk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x2add6a662db0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-wkvrqn6clxhk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x284f14134240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-sf6285dvgkzp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x284f14134240>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-sf6285dvgkzp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 24m 44s branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth n/a passed n/a 0 n/a
1h 1m 52s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
3h 10m 10s branch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 uksouth e2e failed n/a 12
e2e (12)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
00:03:34 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1480002e0090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v6mhpvmjllbc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f4505ea-893f-413a-81c7-aa357a711eaa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f4505ea-893f-413a-81c7-aa357a711eaa",
      "name": "9f4505ea-893f-413a-81c7-aa357a711eaa",
      "status": "Failed",
      "startTime": "2026-10-08T01:19:19.539417138Z",
      "endTime": "2026-10-08T01:48:27.529510506Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-npz5p-fm9t2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-fm9t2 to exist\nMachine cilium-cluster-cilium-np-npz5p-pln66: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-pln66 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-npz5p-fm9t2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-fm9t2 to exist\nMachine cilium-cluster-cilium-np-npz5p-pln66: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-pln66 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1480002e0090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-v6mhpvmjllbc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f4505ea-893f-413a-81c7-aa357a711eaa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f4505ea-893f-413a-81c7-aa357a711eaa",
      "name": "9f4505ea-893f-413a-81c7-aa357a711eaa",
      "status": "Failed",
      "startTime": "2026-10-08T01:19:19.539417138Z",
      "endTime": "2026-10-08T01:48:27.529510506Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-npz5p-fm9t2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-fm9t2 to exist\nMachine cilium-cluster-cilium-np-npz5p-pln66: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-pln66 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-npz5p-fm9t2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-fm9t2 to exist\nMachine cilium-cluster-cilium-np-npz5p-pln66: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-v6mhpvmjllbc--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-npz5p-pln66 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
00:03:34 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vdrwx6" for cluster "cluster-candidate-5-0-vdrwx6"
Unexpected error:
    <*fmt.wrapError | 0xe7465c98060>: 
    failed to create NodePool np-vdrwx6: failed waiting for nodepool="np-vdrwx6" for cluster "cluster-candidate-5-0-vdrwx6" in resourcegroup="rg-candidate-5-0-m92lkknqnj7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5dc4901-b409-4643-aabf-27d9e2910002
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5dc4901-b409-4643-aabf-27d9e2910002",
      "name": "b5dc4901-b409-4643-aabf-27d9e2910002",
      "status": "Failed",
      "startTime": "2026-10-08T01:43:19.047485932Z",
      "endTime": "2026-10-08T02:12:26.297120954Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck to exist\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck to exist\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vdrwx6" for cluster "cluster-candidate-5-0-vdrwx6"
Unexpected error:
    <*fmt.wrapError | 0xe7465c98060>: 
    failed to create NodePool np-vdrwx6: failed waiting for nodepool="np-vdrwx6" for cluster "cluster-candidate-5-0-vdrwx6" in resourcegroup="rg-candidate-5-0-m92lkknqnj7k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5dc4901-b409-4643-aabf-27d9e2910002
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/b5dc4901-b409-4643-aabf-27d9e2910002",
      "name": "b5dc4901-b409-4643-aabf-27d9e2910002",
      "status": "Failed",
      "startTime": "2026-10-08T01:43:19.047485932Z",
      "endTime": "2026-10-08T02:12:26.297120954Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck to exist\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-6b8ck to exist\nMachine f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-m92lkknqnj7k-candidate-vdrwx6--managed/providers/Microsoft.Compute/virtualMachines/f3w7k8p3r3o8h9q-np-vdrwx6-rs755-h28lt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
00:03:34 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm"
Unexpected error:
    <*fmt.wrapError | 0x25efd83de780>: 
    failed to create NodePool np-drrgfm: failed waiting for nodepool="np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm" in resourcegroup="rg-candidate-5-1-7sfzz5wmdjlq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b",
      "name": "c439fc17-ab56-478e-a949-fbf669f84a7b",
      "status": "Failed",
      "startTime": "2026-10-08T01:47:33.608428033Z",
      "endTime": "2026-10-08T02:16:36.341701384Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm"
Unexpected error:
    <*fmt.wrapError | 0x25efd83de780>: 
    failed to create NodePool np-drrgfm: failed waiting for nodepool="np-drrgfm" for cluster "cluster-candidate-5-1-drrgfm" in resourcegroup="rg-candidate-5-1-7sfzz5wmdjlq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/c439fc17-ab56-478e-a949-fbf669f84a7b",
      "name": "c439fc17-ab56-478e-a949-fbf669f84a7b",
      "status": "Failed",
      "startTime": "2026-10-08T01:47:33.608428033Z",
      "endTime": "2026-10-08T02:16:36.341701384Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-nhhtq to exist\nMachine j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-7sfzz5wmdjlq-candidate-drrgfm--managed/providers/Microsoft.Compute/virtualMachines/j1s3k9h1b7u5q9w-np-drrgfm-hkbqc-x9l5r to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
00:03:34 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pcfgpm" for cluster "cluster-candidate-4-23-pcfgpm"
Unexpected error:
    <*fmt.wrapError | 0x3576982986e0>: 
    failed to create NodePool np-pcfgpm: failed waiting for nodepool="np-pcfgpm" for cluster "cluster-candidate-4-23-pcfgpm" in resourcegroup="rg-candidate-4-23-s9jhxk9d7hqt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370524de-fc82-4336-87bc-65ac23926875
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370524de-fc82-4336-87bc-65ac23926875",
      "name": "370524de-fc82-4336-87bc-65ac23926875",
      "status": "Failed",
      "startTime": "2026-10-08T01:50:33.954027938Z",
      "endTime": "2026-10-08T02:19:36.145194017Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r to exist\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r to exist\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pcfgpm" for cluster "cluster-candidate-4-23-pcfgpm"
Unexpected error:
    <*fmt.wrapError | 0x3576982986e0>: 
    failed to create NodePool np-pcfgpm: failed waiting for nodepool="np-pcfgpm" for cluster "cluster-candidate-4-23-pcfgpm" in resourcegroup="rg-candidate-4-23-s9jhxk9d7hqt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370524de-fc82-4336-87bc-65ac23926875
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/370524de-fc82-4336-87bc-65ac23926875",
      "name": "370524de-fc82-4336-87bc-65ac23926875",
      "status": "Failed",
      "startTime": "2026-10-08T01:50:33.954027938Z",
      "endTime": "2026-10-08T02:19:36.145194017Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r to exist\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-grk6r to exist\nMachine z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-s9jhxk9d7hqt-candidate-pcfgpm--managed/providers/Microsoft.Compute/virtualMachines/z9r3j0z3e9p9b2j-np-pcfgpm-6qph8-qxqxf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
00:03:34 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x86f41c2a240>: 
    resource group "rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "name": "1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:56.823039116Z",
      "endTime": "2026-10-08T01:42:56.76971285Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x86f41c2a240>: 
    resource group "rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p8d8rc9ts7fq" to finish deleting: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "name": "1e2cfd55-dd4d-47e4-8c4b-c3e1f95ed759",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:56.823039116Z",
      "endTime": "2026-10-08T01:42:56.76971285Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
00:03:34 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pxdmtf" for cluster "cluster-candidate-4-20-pxdmtf"
Unexpected error:
    <*fmt.wrapError | 0x1f9795690600>: 
    failed to create NodePool np-pxdmtf: failed waiting for nodepool="np-pxdmtf" for cluster "cluster-candidate-4-20-pxdmtf" in resourcegroup="rg-candidate-4-20-pb6dpm6zm8gk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ccf5247-a8a3-4025-b710-f9043bee2e39
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ccf5247-a8a3-4025-b710-f9043bee2e39",
      "name": "7ccf5247-a8a3-4025-b710-f9043bee2e39",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:00.030564256Z",
      "endTime": "2026-10-08T02:11:06.112048987Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn to exist\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn to exist\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-pxdmtf" for cluster "cluster-candidate-4-20-pxdmtf"
Unexpected error:
    <*fmt.wrapError | 0x1f9795690600>: 
    failed to create NodePool np-pxdmtf: failed waiting for nodepool="np-pxdmtf" for cluster "cluster-candidate-4-20-pxdmtf" in resourcegroup="rg-candidate-4-20-pb6dpm6zm8gk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ccf5247-a8a3-4025-b710-f9043bee2e39
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/7ccf5247-a8a3-4025-b710-f9043bee2e39",
      "name": "7ccf5247-a8a3-4025-b710-f9043bee2e39",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:00.030564256Z",
      "endTime": "2026-10-08T02:11:06.112048987Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn to exist\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-dk5fn to exist\nMachine o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-pb6dpm6zm8gk-candidate-pxdmtf--managed/providers/Microsoft.Compute/virtualMachines/o4r5b1i9k3h6r0r-np-pxdmtf-6jkll-pxgpm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
00:03:34 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x170b4dc75dc0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-gp2b86bd5hhp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b07d65c-7304-4507-95e3-59630e971c0f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b07d65c-7304-4507-95e3-59630e971c0f",
      "name": "1b07d65c-7304-4507-95e3-59630e971c0f",
      "status": "Failed",
      "startTime": "2026-10-08T01:21:26.589783314Z",
      "endTime": "2026-10-08T01:50:27.09244579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qt9mw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qt9mw to exist\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qxvfr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qxvfr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qt9mw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qt9mw to exist\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qxvfr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qxvfr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x170b4dc75dc0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-gp2b86bd5hhp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b07d65c-7304-4507-95e3-59630e971c0f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1b07d65c-7304-4507-95e3-59630e971c0f",
      "name": "1b07d65c-7304-4507-95e3-59630e971c0f",
      "status": "Failed",
      "startTime": "2026-10-08T01:21:26.589783314Z",
      "endTime": "2026-10-08T01:50:27.09244579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qt9mw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qt9mw to exist\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qxvfr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qxvfr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qt9mw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qt9mw to exist\nMachine c4d5u2q2v8i5c3s-np-1-8654z-qxvfr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-gp2b86bd5hhp--managed/providers/Microsoft.Compute/virtualMachines/c4d5u2q2v8i5c3s-np-1-8654z-qxvfr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
00:03:34 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xcfd18f3b720>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r4vhjgdwsnjg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "name": "d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "status": "Failed",
      "startTime": "2026-10-08T02:22:41.727850228Z",
      "endTime": "2026-10-08T02:51:47.319042789Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xcfd18f3b720>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-r4vhjgdwsnjg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "name": "d6c35801-94a0-4564-b906-f06f1cdcb28b",
      "status": "Failed",
      "startTime": "2026-10-08T02:22:41.727850228Z",
      "endTime": "2026-10-08T02:51:47.319042789Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-b5742 to exist\nMachine r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-r4vhjgdwsnjg--managed/providers/Microsoft.Compute/virtualMachines/r5d8b8h3a4q7e4f-np-delete-main-fv67r-mqqrr to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
00:03:34 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tdmg5v" for cluster "cluster-candidate-4-21-tdmg5v"
Unexpected error:
    <*fmt.wrapError | 0x3d1b2ad8bcc0>: 
    failed to create NodePool np-tdmg5v: failed waiting for nodepool="np-tdmg5v" for cluster "cluster-candidate-4-21-tdmg5v" in resourcegroup="rg-candidate-4-21-4tbqvrc7sgc8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a2fc-c57b-40ed-870a-b5372a8c7e46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a2fc-c57b-40ed-870a-b5372a8c7e46",
      "name": "1ac5a2fc-c57b-40ed-870a-b5372a8c7e46",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:02.12275582Z",
      "endTime": "2026-10-08T02:11:06.102672717Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc to exist\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc to exist\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tdmg5v" for cluster "cluster-candidate-4-21-tdmg5v"
Unexpected error:
    <*fmt.wrapError | 0x3d1b2ad8bcc0>: 
    failed to create NodePool np-tdmg5v: failed waiting for nodepool="np-tdmg5v" for cluster "cluster-candidate-4-21-tdmg5v" in resourcegroup="rg-candidate-4-21-4tbqvrc7sgc8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a2fc-c57b-40ed-870a-b5372a8c7e46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/1ac5a2fc-c57b-40ed-870a-b5372a8c7e46",
      "name": "1ac5a2fc-c57b-40ed-870a-b5372a8c7e46",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:02.12275582Z",
      "endTime": "2026-10-08T02:11:06.102672717Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc to exist\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-jqcqc to exist\nMachine g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-4tbqvrc7sgc8-candidate-tdmg5v--managed/providers/Microsoft.Compute/virtualMachines/g6c8b7j7z0r2i3j-np-tdmg5v-vtsdm-spkmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
00:03:34 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x3c5e53f64c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-p8759llzs4pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "name": "ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:57.884545997Z",
      "endTime": "2026-10-08T01:47:57.229221309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-n5sb8-kxqvs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-n5sb8-kxqvs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x3c5e53f64c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-p8759llzs4pn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "name": "ddc299f6-13e4-4aac-bd41-1e700b04d5a0",
      "status": "Failed",
      "startTime": "2026-10-08T01:18:57.884545997Z",
      "endTime": "2026-10-08T01:47:57.229221309Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-n5sb8-kxqvs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-n5sb8-kxqvs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-p8759llzs4pn--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-n5sb8-kxqvs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
00:03:34 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5c6cr2" for cluster "cluster-candidate-4-22-5c6cr2"
Unexpected error:
    <*fmt.wrapError | 0xb83ba3deaa0>: 
    failed to create NodePool np-5c6cr2: failed waiting for nodepool="np-5c6cr2" for cluster "cluster-candidate-4-22-5c6cr2" in resourcegroup="rg-candidate-4-22-9rjgc2ft2c5b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62c14f8c-751c-40cf-8cb7-a8c4b8573ef3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62c14f8c-751c-40cf-8cb7-a8c4b8573ef3",
      "name": "62c14f8c-751c-40cf-8cb7-a8c4b8573ef3",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:14.829363977Z",
      "endTime": "2026-10-08T02:11:16.24601066Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm to exist\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm to exist\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5c6cr2" for cluster "cluster-candidate-4-22-5c6cr2"
Unexpected error:
    <*fmt.wrapError | 0xb83ba3deaa0>: 
    failed to create NodePool np-5c6cr2: failed waiting for nodepool="np-5c6cr2" for cluster "cluster-candidate-4-22-5c6cr2" in resourcegroup="rg-candidate-4-22-9rjgc2ft2c5b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62c14f8c-751c-40cf-8cb7-a8c4b8573ef3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/62c14f8c-751c-40cf-8cb7-a8c4b8573ef3",
      "name": "62c14f8c-751c-40cf-8cb7-a8c4b8573ef3",
      "status": "Failed",
      "startTime": "2026-10-08T01:42:14.829363977Z",
      "endTime": "2026-10-08T02:11:16.24601066Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm to exist\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-744gm to exist\nMachine w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-9rjgc2ft2c5b-candidate-5c6cr2--managed/providers/Microsoft.Compute/virtualMachines/w0z5n7c1v2q8l7j-np-5c6cr2-5jkc5-8mnld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
client rate limiter Wait returned an error: context deadline exceeded
00:03:34 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x273477d04048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-zvv7lmwhhzgh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "name": "81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "status": "Failed",
      "startTime": "2026-10-08T01:30:01.277978624Z",
      "endTime": "2026-10-08T01:59:06.873419943Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x273477d04048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-zvv7lmwhhzgh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "name": "81d0dfd2-7968-4d7a-9a69-0ffb6aa5ea78",
      "status": "Failed",
      "startTime": "2026-10-08T01:30:01.277978624Z",
      "endTime": "2026-10-08T01:59:06.873419943Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-2zm5h-ntjmk: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-2zm5h-jnxj4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-zvv7lmwhhzgh--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-2zm5h-jnxj4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred