CIHealth
<
Single Day: Oct 8▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
1h 47m 54s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
 canadacentral e2e failed n/a 5
e2e (5)
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x1bc3bcaaef00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-8cplvshb5svv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x1bc3bcaaef00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-8cplvshb5svv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:38 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x17466b892d50>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-n8q8j5xgxdk4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x17466b892d50>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-n8q8j5xgxdk4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:38 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2e62773595c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-m97nzsnh6v7k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2e62773595c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-m97nzsnh6v7k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:38 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3ea6fe4464b0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rwz8zs2wwdcn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3ea6fe4464b0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rwz8zs2wwdcn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:38 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2d03e6b69a40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-vv42cndqzc2q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2d03e6b69a40>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-vv42cndqzc2q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 12m 24s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 canadacentral n/a passed n/a 0 n/a
31m 24s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 n/a unknown failed n/a 1 non-artifact-backed failure (no junit artifacts)
2h 43m 45s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 centralindia e2e failed n/a 10
e2e (10)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
17:28:22 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x2e7b26aac4c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-fbkb2qz6f9rf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/9afe7411-35e5-4565-a7d7-5d4e665fe58f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/9afe7411-35e5-4565-a7d7-5d4e665fe58f",
      "name": "9afe7411-35e5-4565-a7d7-5d4e665fe58f",
      "status": "Failed",
      "startTime": "2026-10-08T18:03:48.271009954Z",
      "endTime": "2026-10-08T18:32:55.449378647Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n to exist\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-phn9g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-phn9g to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n to exist\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-phn9g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-phn9g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x2e7b26aac4c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-fbkb2qz6f9rf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/9afe7411-35e5-4565-a7d7-5d4e665fe58f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/9afe7411-35e5-4565-a7d7-5d4e665fe58f",
      "name": "9afe7411-35e5-4565-a7d7-5d4e665fe58f",
      "status": "Failed",
      "startTime": "2026-10-08T18:03:48.271009954Z",
      "endTime": "2026-10-08T18:32:55.449378647Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n to exist\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-phn9g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-phn9g to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-lfj4n to exist\nMachine r9j7p3c0y1b3k9z-np-1-44qrg-phn9g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-fbkb2qz6f9rf--managed/providers/Microsoft.Compute/virtualMachines/r9j7p3c0y1b3k9z-np-1-44qrg-phn9g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
17:28:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-cz595b" for cluster "cluster-candidate-4-22-cz595b"
Unexpected error:
    <*fmt.wrapError | 0x18c3c8280a0>: 
    failed to create NodePool np-cz595b: failed waiting for nodepool="np-cz595b" for cluster "cluster-candidate-4-22-cz595b" in resourcegroup="rg-candidate-4-22-vzqcrpffcth2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/7c22ea53-4deb-472f-837d-658a6f8cc25d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/7c22ea53-4deb-472f-837d-658a6f8cc25d",
      "name": "7c22ea53-4deb-472f-837d-658a6f8cc25d",
      "status": "Failed",
      "startTime": "2026-10-08T18:36:25.022726186Z",
      "endTime": "2026-10-08T19:05:25.609844949Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9 to exist\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9 to exist\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-cz595b" for cluster "cluster-candidate-4-22-cz595b"
Unexpected error:
    <*fmt.wrapError | 0x18c3c8280a0>: 
    failed to create NodePool np-cz595b: failed waiting for nodepool="np-cz595b" for cluster "cluster-candidate-4-22-cz595b" in resourcegroup="rg-candidate-4-22-vzqcrpffcth2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/7c22ea53-4deb-472f-837d-658a6f8cc25d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/7c22ea53-4deb-472f-837d-658a6f8cc25d",
      "name": "7c22ea53-4deb-472f-837d-658a6f8cc25d",
      "status": "Failed",
      "startTime": "2026-10-08T18:36:25.022726186Z",
      "endTime": "2026-10-08T19:05:25.609844949Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9 to exist\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-drkw9 to exist\nMachine f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-vzqcrpffcth2-candidate-cz595b--managed/providers/Microsoft.Compute/virtualMachines/f1m0r4q7i9z9q8o-np-cz595b-w6m8q-g48ht to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
17:28:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fn9nq4" for cluster "cluster-candidate-4-21-fn9nq4"
Unexpected error:
    <*fmt.wrapError | 0x1aded3148440>: 
    failed to create NodePool np-fn9nq4: failed waiting for nodepool="np-fn9nq4" for cluster "cluster-candidate-4-21-fn9nq4" in resourcegroup="rg-candidate-4-21-ghsd6g8zdnw5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/70a9d05d-3834-496b-ab23-90c8635291e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/70a9d05d-3834-496b-ab23-90c8635291e9",
      "name": "70a9d05d-3834-496b-ab23-90c8635291e9",
      "status": "Failed",
      "startTime": "2026-10-08T18:35:08.455675057Z",
      "endTime": "2026-10-08T19:04:15.635949535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p to exist\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p to exist\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fn9nq4" for cluster "cluster-candidate-4-21-fn9nq4"
Unexpected error:
    <*fmt.wrapError | 0x1aded3148440>: 
    failed to create NodePool np-fn9nq4: failed waiting for nodepool="np-fn9nq4" for cluster "cluster-candidate-4-21-fn9nq4" in resourcegroup="rg-candidate-4-21-ghsd6g8zdnw5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/70a9d05d-3834-496b-ab23-90c8635291e9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/70a9d05d-3834-496b-ab23-90c8635291e9",
      "name": "70a9d05d-3834-496b-ab23-90c8635291e9",
      "status": "Failed",
      "startTime": "2026-10-08T18:35:08.455675057Z",
      "endTime": "2026-10-08T19:04:15.635949535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p to exist\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-lsc2p to exist\nMachine b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-ghsd6g8zdnw5-candidate-fn9nq4--managed/providers/Microsoft.Compute/virtualMachines/b5c3z4o0q2e6o5q-np-fn9nq4-ttwck-vj8t5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
client rate limiter Wait returned an error: context deadline exceeded
17:28:22 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2f509ecfe3c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bw4mbhgcm827" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "name": "d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "status": "Failed",
      "startTime": "2026-10-08T18:07:59.549473703Z",
      "endTime": "2026-10-08T18:37:05.482678031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2f509ecfe3c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-bw4mbhgcm827" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "name": "d4170a8b-86e1-4b07-aae0-9aa74a31f596",
      "status": "Failed",
      "startTime": "2026-10-08T18:07:59.549473703Z",
      "endTime": "2026-10-08T18:37:05.482678031Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-cvbtc-4pk5m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-4pk5m to exist\nMachine cilium-cluster-cilium-np-cvbtc-mkchh: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-bw4mbhgcm827--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-cvbtc-mkchh to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
17:28:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n"
Unexpected error:
    <*fmt.wrapError | 0x35ae7eca0680>: 
    failed to create NodePool np-ntb88n: failed waiting for nodepool="np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n" in resourcegroup="rg-candidate-5-1-tvlhq8pw8r2j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122",
      "name": "00bc71ab-da38-456c-8cc5-ad615c84b122",
      "status": "Failed",
      "startTime": "2026-10-08T18:45:34.339197477Z",
      "endTime": "2026-10-08T19:14:37.105782951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n"
Unexpected error:
    <*fmt.wrapError | 0x35ae7eca0680>: 
    failed to create NodePool np-ntb88n: failed waiting for nodepool="np-ntb88n" for cluster "cluster-candidate-5-1-ntb88n" in resourcegroup="rg-candidate-5-1-tvlhq8pw8r2j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/00bc71ab-da38-456c-8cc5-ad615c84b122",
      "name": "00bc71ab-da38-456c-8cc5-ad615c84b122",
      "status": "Failed",
      "startTime": "2026-10-08T18:45:34.339197477Z",
      "endTime": "2026-10-08T19:14:37.105782951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-48plr to exist\nMachine z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tvlhq8pw8r2j-candidate-ntb88n--managed/providers/Microsoft.Compute/virtualMachines/z8r5l5c1h2t0o8m-np-ntb88n-lmxzn-hfnpk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
client rate limiter Wait returned an error: context deadline exceeded
17:28:22 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x3c50440e0018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-nkr4c2vp869x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e",
      "name": "5de3f7d0-158f-4a7e-a885-89292635394e",
      "status": "Failed",
      "startTime": "2026-10-08T18:08:57.323157264Z",
      "endTime": "2026-10-08T18:38:05.479695963Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x3c50440e0018>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-nkr4c2vp869x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/5de3f7d0-158f-4a7e-a885-89292635394e",
      "name": "5de3f7d0-158f-4a7e-a885-89292635394e",
      "status": "Failed",
      "startTime": "2026-10-08T18:08:57.323157264Z",
      "endTime": "2026-10-08T18:38:05.479695963Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-ffrmf-nnf5q: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-ffrmf-shgcl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-nkr4c2vp869x--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-ffrmf-shgcl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
17:28:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dlrjwb" for cluster "cluster-candidate-4-23-dlrjwb"
Unexpected error:
    <*fmt.wrapError | 0x2b65d70d0400>: 
    failed to create NodePool np-dlrjwb: failed waiting for nodepool="np-dlrjwb" for cluster "cluster-candidate-4-23-dlrjwb" in resourcegroup="rg-candidate-4-23-sdtvdwbjbxfr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/79fc773d-2a37-4106-8f63-f194f928388d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/79fc773d-2a37-4106-8f63-f194f928388d",
      "name": "79fc773d-2a37-4106-8f63-f194f928388d",
      "status": "Failed",
      "startTime": "2026-10-08T18:36:39.156679392Z",
      "endTime": "2026-10-08T19:05:45.645587063Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z to exist\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z to exist\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-dlrjwb" for cluster "cluster-candidate-4-23-dlrjwb"
Unexpected error:
    <*fmt.wrapError | 0x2b65d70d0400>: 
    failed to create NodePool np-dlrjwb: failed waiting for nodepool="np-dlrjwb" for cluster "cluster-candidate-4-23-dlrjwb" in resourcegroup="rg-candidate-4-23-sdtvdwbjbxfr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/79fc773d-2a37-4106-8f63-f194f928388d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/79fc773d-2a37-4106-8f63-f194f928388d",
      "name": "79fc773d-2a37-4106-8f63-f194f928388d",
      "status": "Failed",
      "startTime": "2026-10-08T18:36:39.156679392Z",
      "endTime": "2026-10-08T19:05:45.645587063Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z to exist\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-96k5z to exist\nMachine w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sdtvdwbjbxfr-candidate-dlrjwb--managed/providers/Microsoft.Compute/virtualMachines/w2l2k7p1t2a1o2w-np-dlrjwb-79crd-vcm8b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
17:28:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c"
Unexpected error:
    <*fmt.wrapError | 0x553d3fef80>: 
    failed to create NodePool np-kn7h4c: failed waiting for nodepool="np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c" in resourcegroup="rg-candidate-5-0-5tp4pjp8hk9v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "name": "b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "status": "Failed",
      "startTime": "2026-10-08T18:41:12.213982763Z",
      "endTime": "2026-10-08T19:10:15.712133624Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c"
Unexpected error:
    <*fmt.wrapError | 0x553d3fef80>: 
    failed to create NodePool np-kn7h4c: failed waiting for nodepool="np-kn7h4c" for cluster "cluster-candidate-5-0-kn7h4c" in resourcegroup="rg-candidate-5-0-5tp4pjp8hk9v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "name": "b83f80b7-a3b4-4c91-9ea3-c3b3373ebcbb",
      "status": "Failed",
      "startTime": "2026-10-08T18:41:12.213982763Z",
      "endTime": "2026-10-08T19:10:15.712133624Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-bs7p4 to exist\nMachine b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-5tp4pjp8hk9v-candidate-kn7h4c--managed/providers/Microsoft.Compute/virtualMachines/b8y3z9m8y0d2g4x-np-kn7h4c-z99qt-pxbmm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
17:28:22 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm"
Unexpected error:
    <*fmt.wrapError | 0x3108b67364e0>: 
    failed to create NodePool np-8mqkrm: failed waiting for nodepool="np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm" in resourcegroup="rg-candidate-4-20-xmx872mb68nv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "name": "827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "status": "Failed",
      "startTime": "2026-10-08T18:32:58.020047255Z",
      "endTime": "2026-10-08T19:02:05.605513431Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm"
Unexpected error:
    <*fmt.wrapError | 0x3108b67364e0>: 
    failed to create NodePool np-8mqkrm: failed waiting for nodepool="np-8mqkrm" for cluster "cluster-candidate-4-20-8mqkrm" in resourcegroup="rg-candidate-4-20-xmx872mb68nv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "name": "827a6af5-91f2-4a87-8008-85cec7a38eaf",
      "status": "Failed",
      "startTime": "2026-10-08T18:32:58.020047255Z",
      "endTime": "2026-10-08T19:02:05.605513431Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-cv7mc to exist\nMachine k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-xmx872mb68nv-candidate-8mqkrm--managed/providers/Microsoft.Compute/virtualMachines/k5p6u6p3d8e6w7b-np-8mqkrm-lk5dd-wvc7z to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
17:28:22 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x237e16e43180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-q28hc5z2kwsh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/6c8ad6b0-3620-4b89-8c2d-08b0866db056
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/6c8ad6b0-3620-4b89-8c2d-08b0866db056",
      "name": "6c8ad6b0-3620-4b89-8c2d-08b0866db056",
      "status": "Failed",
      "startTime": "2026-10-08T18:03:48.204928396Z",
      "endTime": "2026-10-08T18:32:55.458196493Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-j8vmw-6lgjc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-q28hc5z2kwsh--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-j8vmw-6lgjc to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-j8vmw-6lgjc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-q28hc5z2kwsh--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-j8vmw-6lgjc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x237e16e43180>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-q28hc5z2kwsh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/6c8ad6b0-3620-4b89-8c2d-08b0866db056
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralindia/hcpOperationStatuses/6c8ad6b0-3620-4b89-8c2d-08b0866db056",
      "name": "6c8ad6b0-3620-4b89-8c2d-08b0866db056",
      "status": "Failed",
      "startTime": "2026-10-08T18:03:48.204928396Z",
      "endTime": "2026-10-08T18:32:55.458196493Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-j8vmw-6lgjc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-q28hc5z2kwsh--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-j8vmw-6lgjc to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-j8vmw-6lgjc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-q28hc5z2kwsh--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-j8vmw-6lgjc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 42m 37s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth e2e failed n/a 11
e2e (11)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
13:52:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qzm2gk" for cluster "cluster-candidate-4-21-qzm2gk"
Unexpected error:
    <*fmt.wrapError | 0x39e342643720>: 
    failed to create NodePool np-qzm2gk: failed waiting for nodepool="np-qzm2gk" for cluster "cluster-candidate-4-21-qzm2gk" in resourcegroup="rg-candidate-4-21-t5wksgdrrckp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b",
      "name": "afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b",
      "status": "Failed",
      "startTime": "2026-10-08T14:53:33.939084658Z",
      "endTime": "2026-10-08T15:22:35.324122041Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t to exist\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t to exist\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-qzm2gk" for cluster "cluster-candidate-4-21-qzm2gk"
Unexpected error:
    <*fmt.wrapError | 0x39e342643720>: 
    failed to create NodePool np-qzm2gk: failed waiting for nodepool="np-qzm2gk" for cluster "cluster-candidate-4-21-qzm2gk" in resourcegroup="rg-candidate-4-21-t5wksgdrrckp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b",
      "name": "afe11fbe-72f7-4e4f-b45c-eb8ad99b3d2b",
      "status": "Failed",
      "startTime": "2026-10-08T14:53:33.939084658Z",
      "endTime": "2026-10-08T15:22:35.324122041Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t to exist\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-6tq9t to exist\nMachine w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-t5wksgdrrckp-candidate-qzm2gk--managed/providers/Microsoft.Compute/virtualMachines/w9e3g8e6x8h0t9p-np-qzm2gk-lblkl-jlq5b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
13:52:10 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x371874876980>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-ftnkcdsss4wg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/883d0d0d-c572-41f2-9f82-7d61a050db28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/883d0d0d-c572-41f2-9f82-7d61a050db28",
      "name": "883d0d0d-c572-41f2-9f82-7d61a050db28",
      "status": "Failed",
      "startTime": "2026-10-08T14:27:29.613460957Z",
      "endTime": "2026-10-08T14:56:34.939890509Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv to exist\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv to exist\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x371874876980>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-ftnkcdsss4wg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/883d0d0d-c572-41f2-9f82-7d61a050db28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/883d0d0d-c572-41f2-9f82-7d61a050db28",
      "name": "883d0d0d-c572-41f2-9f82-7d61a050db28",
      "status": "Failed",
      "startTime": "2026-10-08T14:27:29.613460957Z",
      "endTime": "2026-10-08T14:56:34.939890509Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv to exist\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-g7kbv to exist\nMachine s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-ftnkcdsss4wg--managed/providers/Microsoft.Compute/virtualMachines/s6j5k7l8f8s0p3k-np-1-5bsqt-h5mb4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
client rate limiter Wait returned an error: context deadline exceeded
13:52:10 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x6335433a090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-tpwrtzjcnmjq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "name": "3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "status": "Failed",
      "startTime": "2026-10-08T14:27:33.571990767Z",
      "endTime": "2026-10-08T14:56:35.04864797Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-8swjn-292k4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-8swjn-292k4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x6335433a090>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-tpwrtzjcnmjq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "name": "3feda0f9-93b1-4baa-b95d-bcd59b54cae3",
      "status": "Failed",
      "startTime": "2026-10-08T14:27:33.571990767Z",
      "endTime": "2026-10-08T14:56:35.04864797Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-8swjn-292k4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-8swjn-b2vlb: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-8swjn-292k4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-tpwrtzjcnmjq--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-8swjn-292k4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
13:52:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ctg8q4" for cluster "cluster-candidate-4-23-ctg8q4"
Unexpected error:
    <*fmt.wrapError | 0x3ac3b36042c0>: 
    failed to create NodePool np-ctg8q4: failed waiting for nodepool="np-ctg8q4" for cluster "cluster-candidate-4-23-ctg8q4" in resourcegroup="rg-candidate-4-23-z87lsqfql56h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4467076c-8824-4b5f-8e60-3977a0fe20c4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4467076c-8824-4b5f-8e60-3977a0fe20c4",
      "name": "4467076c-8824-4b5f-8e60-3977a0fe20c4",
      "status": "Failed",
      "startTime": "2026-10-08T14:57:55.181740971Z",
      "endTime": "2026-10-08T15:27:00.08772636Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc to exist\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc to exist\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-ctg8q4" for cluster "cluster-candidate-4-23-ctg8q4"
Unexpected error:
    <*fmt.wrapError | 0x3ac3b36042c0>: 
    failed to create NodePool np-ctg8q4: failed waiting for nodepool="np-ctg8q4" for cluster "cluster-candidate-4-23-ctg8q4" in resourcegroup="rg-candidate-4-23-z87lsqfql56h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4467076c-8824-4b5f-8e60-3977a0fe20c4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4467076c-8824-4b5f-8e60-3977a0fe20c4",
      "name": "4467076c-8824-4b5f-8e60-3977a0fe20c4",
      "status": "Failed",
      "startTime": "2026-10-08T14:57:55.181740971Z",
      "endTime": "2026-10-08T15:27:00.08772636Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc to exist\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-gdptc to exist\nMachine l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-z87lsqfql56h-candidate-ctg8q4--managed/providers/Microsoft.Compute/virtualMachines/l7q9a4u1c8n5z2a-np-ctg8q4-cvxjg-kzdmt to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
13:52:10 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3a6cffe3f060>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-jct5pzjh8mxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "name": "b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "status": "Failed",
      "startTime": "2026-10-08T15:48:06.632742492Z",
      "endTime": "2026-10-08T16:17:11.778582607Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3a6cffe3f060>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-jct5pzjh8mxm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "name": "b64e41d5-253f-4190-baa0-1b895cdf4caf",
      "status": "Failed",
      "startTime": "2026-10-08T15:48:06.632742492Z",
      "endTime": "2026-10-08T16:17:11.778582607Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-6hkbw to exist\nMachine x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-jct5pzjh8mxm--managed/providers/Microsoft.Compute/virtualMachines/x0a2x4h1b1y4w4v-np-delete-main-8n5mr-wzs5x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
13:52:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-24nxrk" for cluster "cluster-candidate-5-1-24nxrk"
Unexpected error:
    <*fmt.wrapError | 0x36b6bf7fec00>: 
    failed to create NodePool np-24nxrk: failed waiting for nodepool="np-24nxrk" for cluster "cluster-candidate-5-1-24nxrk" in resourcegroup="rg-candidate-5-1-74n6vxnsbzln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a208b4c2-0cec-4a7a-8f77-7140b3a3f092
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a208b4c2-0cec-4a7a-8f77-7140b3a3f092",
      "name": "a208b4c2-0cec-4a7a-8f77-7140b3a3f092",
      "status": "Failed",
      "startTime": "2026-10-08T14:58:11.889421655Z",
      "endTime": "2026-10-08T15:27:20.151153667Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm to exist\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm to exist\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-24nxrk" for cluster "cluster-candidate-5-1-24nxrk"
Unexpected error:
    <*fmt.wrapError | 0x36b6bf7fec00>: 
    failed to create NodePool np-24nxrk: failed waiting for nodepool="np-24nxrk" for cluster "cluster-candidate-5-1-24nxrk" in resourcegroup="rg-candidate-5-1-74n6vxnsbzln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a208b4c2-0cec-4a7a-8f77-7140b3a3f092
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a208b4c2-0cec-4a7a-8f77-7140b3a3f092",
      "name": "a208b4c2-0cec-4a7a-8f77-7140b3a3f092",
      "status": "Failed",
      "startTime": "2026-10-08T14:58:11.889421655Z",
      "endTime": "2026-10-08T15:27:20.151153667Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm to exist\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-4wfrm to exist\nMachine p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-74n6vxnsbzln-candidate-24nxrk--managed/providers/Microsoft.Compute/virtualMachines/p6j0l4e0q2t9w1a-np-24nxrk-9749x-7hqz5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
13:52:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fgmnm8" for cluster "cluster-candidate-4-22-fgmnm8"
Unexpected error:
    <*fmt.wrapError | 0x2ad692bfa2c0>: 
    failed to create NodePool np-fgmnm8: failed waiting for nodepool="np-fgmnm8" for cluster "cluster-candidate-4-22-fgmnm8" in resourcegroup="rg-candidate-4-22-t7czqbb9gdw6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df2fc222-a282-4c8a-885f-6afa0d259ffa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df2fc222-a282-4c8a-885f-6afa0d259ffa",
      "name": "df2fc222-a282-4c8a-885f-6afa0d259ffa",
      "status": "Failed",
      "startTime": "2026-10-08T15:00:21.098278607Z",
      "endTime": "2026-10-08T15:29:30.220139004Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp to exist\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp to exist\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fgmnm8" for cluster "cluster-candidate-4-22-fgmnm8"
Unexpected error:
    <*fmt.wrapError | 0x2ad692bfa2c0>: 
    failed to create NodePool np-fgmnm8: failed waiting for nodepool="np-fgmnm8" for cluster "cluster-candidate-4-22-fgmnm8" in resourcegroup="rg-candidate-4-22-t7czqbb9gdw6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df2fc222-a282-4c8a-885f-6afa0d259ffa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df2fc222-a282-4c8a-885f-6afa0d259ffa",
      "name": "df2fc222-a282-4c8a-885f-6afa0d259ffa",
      "status": "Failed",
      "startTime": "2026-10-08T15:00:21.098278607Z",
      "endTime": "2026-10-08T15:29:30.220139004Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp to exist\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-fpkgp to exist\nMachine o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-t7czqbb9gdw6-candidate-fgmnm8--managed/providers/Microsoft.Compute/virtualMachines/o5a0g0z5v8y7m7t-np-fgmnm8-mjz2d-l6qc5 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
13:52:10 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x248495222160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-2s5lcfmgt854" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/93a34e6e-477c-4e65-9e08-527e17028921
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/93a34e6e-477c-4e65-9e08-527e17028921",
      "name": "93a34e6e-477c-4e65-9e08-527e17028921",
      "status": "Failed",
      "startTime": "2026-10-08T14:23:16.915326226Z",
      "endTime": "2026-10-08T14:52:17.249683042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-xxpw5-lhcfb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2s5lcfmgt854--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-xxpw5-lhcfb to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-xxpw5-lhcfb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2s5lcfmgt854--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-xxpw5-lhcfb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x248495222160>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-2s5lcfmgt854" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/93a34e6e-477c-4e65-9e08-527e17028921
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/93a34e6e-477c-4e65-9e08-527e17028921",
      "name": "93a34e6e-477c-4e65-9e08-527e17028921",
      "status": "Failed",
      "startTime": "2026-10-08T14:23:16.915326226Z",
      "endTime": "2026-10-08T14:52:17.249683042Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-xxpw5-lhcfb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2s5lcfmgt854--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-xxpw5-lhcfb to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-xxpw5-lhcfb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-2s5lcfmgt854--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-xxpw5-lhcfb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
13:52:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fsgggm" for cluster "cluster-candidate-4-20-fsgggm"
Unexpected error:
    <*fmt.wrapError | 0x1c37b4b9e200>: 
    failed to create NodePool np-fsgggm: failed waiting for nodepool="np-fsgggm" for cluster "cluster-candidate-4-20-fsgggm" in resourcegroup="rg-candidate-4-20-rz65zsts87v9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/240398ab-2ae4-4b7b-a355-04e9e2139068
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/240398ab-2ae4-4b7b-a355-04e9e2139068",
      "name": "240398ab-2ae4-4b7b-a355-04e9e2139068",
      "status": "Failed",
      "startTime": "2026-10-08T14:57:34.48901399Z",
      "endTime": "2026-10-08T15:26:35.280704579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn to exist\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn to exist\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-fsgggm" for cluster "cluster-candidate-4-20-fsgggm"
Unexpected error:
    <*fmt.wrapError | 0x1c37b4b9e200>: 
    failed to create NodePool np-fsgggm: failed waiting for nodepool="np-fsgggm" for cluster "cluster-candidate-4-20-fsgggm" in resourcegroup="rg-candidate-4-20-rz65zsts87v9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/240398ab-2ae4-4b7b-a355-04e9e2139068
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/240398ab-2ae4-4b7b-a355-04e9e2139068",
      "name": "240398ab-2ae4-4b7b-a355-04e9e2139068",
      "status": "Failed",
      "startTime": "2026-10-08T14:57:34.48901399Z",
      "endTime": "2026-10-08T15:26:35.280704579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn to exist\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-7xmfn to exist\nMachine w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rz65zsts87v9-candidate-fsgggm--managed/providers/Microsoft.Compute/virtualMachines/w4t7a3t7b9u9j2z-np-fsgggm-bz2bl-rjxzc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
13:52:10 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2a3a9b00e6d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-hxkght4ww89l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1026664-b5ca-42f5-adc7-85aaeef21d81
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1026664-b5ca-42f5-adc7-85aaeef21d81",
      "name": "f1026664-b5ca-42f5-adc7-85aaeef21d81",
      "status": "Failed",
      "startTime": "2026-10-08T14:28:38.920968051Z",
      "endTime": "2026-10-08T14:57:44.886200601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-8bbcc-kjgxq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-kjgxq to exist\nMachine cilium-cluster-cilium-np-8bbcc-krtkf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-krtkf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-8bbcc-kjgxq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-kjgxq to exist\nMachine cilium-cluster-cilium-np-8bbcc-krtkf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-krtkf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2a3a9b00e6d8>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-hxkght4ww89l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1026664-b5ca-42f5-adc7-85aaeef21d81
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1026664-b5ca-42f5-adc7-85aaeef21d81",
      "name": "f1026664-b5ca-42f5-adc7-85aaeef21d81",
      "status": "Failed",
      "startTime": "2026-10-08T14:28:38.920968051Z",
      "endTime": "2026-10-08T14:57:44.886200601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-8bbcc-kjgxq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-kjgxq to exist\nMachine cilium-cluster-cilium-np-8bbcc-krtkf: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-krtkf to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-8bbcc-kjgxq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-kjgxq to exist\nMachine cilium-cluster-cilium-np-8bbcc-krtkf: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-hxkght4ww89l--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-8bbcc-krtkf to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
13:52:10 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-c7f6vs" for cluster "cluster-candidate-5-0-c7f6vs"
Unexpected error:
    <*fmt.wrapError | 0xa91b8f312a0>: 
    failed to create NodePool np-c7f6vs: failed waiting for nodepool="np-c7f6vs" for cluster "cluster-candidate-5-0-c7f6vs" in resourcegroup="rg-candidate-5-0-mlpng9ckwtsq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/630087f9-3e91-4cc3-95cc-3a0875463d79
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/630087f9-3e91-4cc3-95cc-3a0875463d79",
      "name": "630087f9-3e91-4cc3-95cc-3a0875463d79",
      "status": "Failed",
      "startTime": "2026-10-08T14:56:46.681105188Z",
      "endTime": "2026-10-08T15:25:55.347852762Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr to exist\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr to exist\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-c7f6vs" for cluster "cluster-candidate-5-0-c7f6vs"
Unexpected error:
    <*fmt.wrapError | 0xa91b8f312a0>: 
    failed to create NodePool np-c7f6vs: failed waiting for nodepool="np-c7f6vs" for cluster "cluster-candidate-5-0-c7f6vs" in resourcegroup="rg-candidate-5-0-mlpng9ckwtsq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/630087f9-3e91-4cc3-95cc-3a0875463d79
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/630087f9-3e91-4cc3-95cc-3a0875463d79",
      "name": "630087f9-3e91-4cc3-95cc-3a0875463d79",
      "status": "Failed",
      "startTime": "2026-10-08T14:56:46.681105188Z",
      "endTime": "2026-10-08T15:25:55.347852762Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr to exist\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-665jr to exist\nMachine c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mlpng9ckwtsq-candidate-c7f6vs--managed/providers/Microsoft.Compute/virtualMachines/c3n8k1n1b0g8r5z-np-c7f6vs-2zsgf-lnfgz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 55m 7s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 brazilsouth e2e failed n/a 11
e2e (11)
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
03:46:27 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-cmn25n" for cluster "cluster-candidate-5-0-cmn25n"
Unexpected error:
    <*fmt.wrapError | 0x19e2b2505020>: 
    failed to create NodePool np-cmn25n: failed waiting for nodepool="np-cmn25n" for cluster "cluster-candidate-5-0-cmn25n" in resourcegroup="rg-candidate-5-0-lhc5fj69t9pm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6879bb7c-c254-4641-ba70-7b41bb04b080
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6879bb7c-c254-4641-ba70-7b41bb04b080",
      "name": "6879bb7c-c254-4641-ba70-7b41bb04b080",
      "status": "Failed",
      "startTime": "2026-10-08T04:52:10.585592043Z",
      "endTime": "2026-10-08T05:21:19.889088931Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l to exist\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l to exist\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-cmn25n" for cluster "cluster-candidate-5-0-cmn25n"
Unexpected error:
    <*fmt.wrapError | 0x19e2b2505020>: 
    failed to create NodePool np-cmn25n: failed waiting for nodepool="np-cmn25n" for cluster "cluster-candidate-5-0-cmn25n" in resourcegroup="rg-candidate-5-0-lhc5fj69t9pm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6879bb7c-c254-4641-ba70-7b41bb04b080
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6879bb7c-c254-4641-ba70-7b41bb04b080",
      "name": "6879bb7c-c254-4641-ba70-7b41bb04b080",
      "status": "Failed",
      "startTime": "2026-10-08T04:52:10.585592043Z",
      "endTime": "2026-10-08T05:21:19.889088931Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l to exist\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-c2h6l to exist\nMachine r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-lhc5fj69t9pm-candidate-cmn25n--managed/providers/Microsoft.Compute/virtualMachines/r6x9q5o3w7j6w0k-np-cmn25n-kb9xb-lcbf7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
03:46:27 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1f97b95fe138>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-zl88ncfz4jp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a2491cc-5c5e-4c96-b7d8-ff0a13549b34
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a2491cc-5c5e-4c96-b7d8-ff0a13549b34",
      "name": "4a2491cc-5c5e-4c96-b7d8-ff0a13549b34",
      "status": "Failed",
      "startTime": "2026-10-08T04:18:50.555017189Z",
      "endTime": "2026-10-08T04:47:59.745762657Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-n67fb-47fdq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-47fdq to exist\nMachine cilium-cluster-cilium-np-n67fb-jqb5d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-jqb5d to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-n67fb-47fdq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-47fdq to exist\nMachine cilium-cluster-cilium-np-n67fb-jqb5d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-jqb5d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x1f97b95fe138>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-zl88ncfz4jp4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a2491cc-5c5e-4c96-b7d8-ff0a13549b34
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4a2491cc-5c5e-4c96-b7d8-ff0a13549b34",
      "name": "4a2491cc-5c5e-4c96-b7d8-ff0a13549b34",
      "status": "Failed",
      "startTime": "2026-10-08T04:18:50.555017189Z",
      "endTime": "2026-10-08T04:47:59.745762657Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-n67fb-47fdq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-47fdq to exist\nMachine cilium-cluster-cilium-np-n67fb-jqb5d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-jqb5d to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-n67fb-47fdq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-47fdq to exist\nMachine cilium-cluster-cilium-np-n67fb-jqb5d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-zl88ncfz4jp4--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-n67fb-jqb5d to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
03:46:27 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x228c1da4a2e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-tff6vqrfwntq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/40cea923-3d28-46b1-b377-958e6c7d32be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/40cea923-3d28-46b1-b377-958e6c7d32be",
      "name": "40cea923-3d28-46b1-b377-958e6c7d32be",
      "status": "Failed",
      "startTime": "2026-10-08T04:17:53.289622119Z",
      "endTime": "2026-10-08T04:46:59.632942955Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-7vfqr-gvgx9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-tff6vqrfwntq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-7vfqr-gvgx9 to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-7vfqr-gvgx9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-tff6vqrfwntq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-7vfqr-gvgx9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x228c1da4a2e0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-tff6vqrfwntq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/40cea923-3d28-46b1-b377-958e6c7d32be
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/40cea923-3d28-46b1-b377-958e6c7d32be",
      "name": "40cea923-3d28-46b1-b377-958e6c7d32be",
      "status": "Failed",
      "startTime": "2026-10-08T04:17:53.289622119Z",
      "endTime": "2026-10-08T04:46:59.632942955Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-7vfqr-gvgx9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-tff6vqrfwntq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-7vfqr-gvgx9 to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-7vfqr-gvgx9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-tff6vqrfwntq--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-7vfqr-gvgx9 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
03:46:27 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xbf2bf37c078>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rgbgvj2hmvgv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa",
      "name": "5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa",
      "status": "Failed",
      "startTime": "2026-10-08T04:21:09.226625204Z",
      "endTime": "2026-10-08T04:50:09.941476385Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-gd5q9-ck6dg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-gd5q9-kqsvs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rgbgvj2hmvgv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-gd5q9-kqsvs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-gd5q9-ck6dg: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-gd5q9-kqsvs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rgbgvj2hmvgv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-gd5q9-kqsvs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xbf2bf37c078>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-rgbgvj2hmvgv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa",
      "name": "5b9ce1ac-a1fe-4426-8a3c-1594d8368aaa",
      "status": "Failed",
      "startTime": "2026-10-08T04:21:09.226625204Z",
      "endTime": "2026-10-08T04:50:09.941476385Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-gd5q9-ck6dg: InspectionFailed: Waiting for AzureMachine to report spec.providerID\nMachine agg-cluster-agg-np-a-gd5q9-kqsvs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rgbgvj2hmvgv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-gd5q9-kqsvs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-gd5q9-ck6dg: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-gd5q9-kqsvs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rgbgvj2hmvgv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-gd5q9-kqsvs to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
03:46:27 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2wf5g9" for cluster "cluster-candidate-5-1-2wf5g9"
Unexpected error:
    <*fmt.wrapError | 0x28a272d90ba0>: 
    failed to create NodePool np-2wf5g9: failed waiting for nodepool="np-2wf5g9" for cluster "cluster-candidate-5-1-2wf5g9" in resourcegroup="rg-candidate-5-1-tpzmdkt924rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b",
      "name": "acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b",
      "status": "Failed",
      "startTime": "2026-10-08T04:52:41.296917226Z",
      "endTime": "2026-10-08T05:21:49.770952812Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl to exist\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl to exist\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-2wf5g9" for cluster "cluster-candidate-5-1-2wf5g9"
Unexpected error:
    <*fmt.wrapError | 0x28a272d90ba0>: 
    failed to create NodePool np-2wf5g9: failed waiting for nodepool="np-2wf5g9" for cluster "cluster-candidate-5-1-2wf5g9" in resourcegroup="rg-candidate-5-1-tpzmdkt924rr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b",
      "name": "acf03b6d-6e69-4e4c-b3e4-ac3e6038ad4b",
      "status": "Failed",
      "startTime": "2026-10-08T04:52:41.296917226Z",
      "endTime": "2026-10-08T05:21:49.770952812Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl to exist\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-hr2rl to exist\nMachine v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-tpzmdkt924rr-candidate-2wf5g9--managed/providers/Microsoft.Compute/virtualMachines/v2e2m0f1q9f1d4a-np-2wf5g9-lbj8r-vj5nq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
03:46:27 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5fp562" for cluster "cluster-candidate-4-22-5fp562"
Unexpected error:
    <*fmt.wrapError | 0x1acc4afec200>: 
    failed to create NodePool np-5fp562: failed waiting for nodepool="np-5fp562" for cluster "cluster-candidate-4-22-5fp562" in resourcegroup="rg-candidate-4-22-x7bk4qfg72wt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df0ac87d-a82c-40b8-81b5-d723efa85d74
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df0ac87d-a82c-40b8-81b5-d723efa85d74",
      "name": "df0ac87d-a82c-40b8-81b5-d723efa85d74",
      "status": "Failed",
      "startTime": "2026-10-08T04:49:21.745083143Z",
      "endTime": "2026-10-08T05:18:29.866533862Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt to exist\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt to exist\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5fp562" for cluster "cluster-candidate-4-22-5fp562"
Unexpected error:
    <*fmt.wrapError | 0x1acc4afec200>: 
    failed to create NodePool np-5fp562: failed waiting for nodepool="np-5fp562" for cluster "cluster-candidate-4-22-5fp562" in resourcegroup="rg-candidate-4-22-x7bk4qfg72wt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df0ac87d-a82c-40b8-81b5-d723efa85d74
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/df0ac87d-a82c-40b8-81b5-d723efa85d74",
      "name": "df0ac87d-a82c-40b8-81b5-d723efa85d74",
      "status": "Failed",
      "startTime": "2026-10-08T04:49:21.745083143Z",
      "endTime": "2026-10-08T05:18:29.866533862Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt to exist\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-qqbdt to exist\nMachine v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-x7bk4qfg72wt-candidate-5fp562--managed/providers/Microsoft.Compute/virtualMachines/v1g0k7c0m8d2y7v-np-5fp562-585r2-t7452 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
03:46:27 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x12d67242a00>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdb7hgvdsm8w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "name": "1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "status": "Failed",
      "startTime": "2026-10-08T05:42:55.903350034Z",
      "endTime": "2026-10-08T06:12:00.742498982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x12d67242a00>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wdb7hgvdsm8w" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "name": "1e7bf622-4bc7-4a43-8991-89c9b6f6d374",
      "status": "Failed",
      "startTime": "2026-10-08T05:42:55.903350034Z",
      "endTime": "2026-10-08T06:12:00.742498982Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-p97bl to exist\nMachine n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wdb7hgvdsm8w--managed/providers/Microsoft.Compute/virtualMachines/n3e8p9e9i8i5i1r-np-deletable-fbchf-rqjc7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
03:46:27 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3cad2d10c0a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4xktww2hxb7t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7eed3c62-0b0c-4aed-9a04-226a72003a56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7eed3c62-0b0c-4aed-9a04-226a72003a56",
      "name": "7eed3c62-0b0c-4aed-9a04-226a72003a56",
      "status": "Failed",
      "startTime": "2026-10-08T04:19:51.649780345Z",
      "endTime": "2026-10-08T04:48:59.779189852Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7 to exist\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7 to exist\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x3cad2d10c0a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-4xktww2hxb7t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7eed3c62-0b0c-4aed-9a04-226a72003a56
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7eed3c62-0b0c-4aed-9a04-226a72003a56",
      "name": "7eed3c62-0b0c-4aed-9a04-226a72003a56",
      "status": "Failed",
      "startTime": "2026-10-08T04:19:51.649780345Z",
      "endTime": "2026-10-08T04:48:59.779189852Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7 to exist\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-rz6v7 to exist\nMachine b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-4xktww2hxb7t--managed/providers/Microsoft.Compute/virtualMachines/b3j0e6f1d8b5y0c-np-1-fs9xj-t8fpj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
03:46:27 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nm4ft4" for cluster "cluster-candidate-4-20-nm4ft4"
Unexpected error:
    <*fmt.wrapError | 0x1e5feac2cf20>: 
    failed to create NodePool np-nm4ft4: failed waiting for nodepool="np-nm4ft4" for cluster "cluster-candidate-4-20-nm4ft4" in resourcegroup="rg-candidate-4-20-gwp4bdzvdkfn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7d0b1c33-9150-4065-8244-327119876e65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7d0b1c33-9150-4065-8244-327119876e65",
      "name": "7d0b1c33-9150-4065-8244-327119876e65",
      "status": "Failed",
      "startTime": "2026-10-08T04:47:03.009170999Z",
      "endTime": "2026-10-08T05:16:09.831739591Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw to exist\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw to exist\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-nm4ft4" for cluster "cluster-candidate-4-20-nm4ft4"
Unexpected error:
    <*fmt.wrapError | 0x1e5feac2cf20>: 
    failed to create NodePool np-nm4ft4: failed waiting for nodepool="np-nm4ft4" for cluster "cluster-candidate-4-20-nm4ft4" in resourcegroup="rg-candidate-4-20-gwp4bdzvdkfn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7d0b1c33-9150-4065-8244-327119876e65
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/7d0b1c33-9150-4065-8244-327119876e65",
      "name": "7d0b1c33-9150-4065-8244-327119876e65",
      "status": "Failed",
      "startTime": "2026-10-08T04:47:03.009170999Z",
      "endTime": "2026-10-08T05:16:09.831739591Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw to exist\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-c72fw to exist\nMachine l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-gwp4bdzvdkfn-candidate-nm4ft4--managed/providers/Microsoft.Compute/virtualMachines/l0f1y9g7w2g8h1t-np-nm4ft4-rnsd9-vzhb4 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
03:46:27 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kjsxrn" for cluster "cluster-candidate-4-23-kjsxrn"
Unexpected error:
    <*fmt.wrapError | 0x27ed88bbf980>: 
    failed to create NodePool np-kjsxrn: failed waiting for nodepool="np-kjsxrn" for cluster "cluster-candidate-4-23-kjsxrn" in resourcegroup="rg-candidate-4-23-9blkbrrd87j2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ea16240d-6235-4ea8-8e63-3873031d6d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ea16240d-6235-4ea8-8e63-3873031d6d51",
      "name": "ea16240d-6235-4ea8-8e63-3873031d6d51",
      "status": "Failed",
      "startTime": "2026-10-08T04:54:34.518182875Z",
      "endTime": "2026-10-08T05:23:35.635646755Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2 to exist\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2 to exist\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-kjsxrn" for cluster "cluster-candidate-4-23-kjsxrn"
Unexpected error:
    <*fmt.wrapError | 0x27ed88bbf980>: 
    failed to create NodePool np-kjsxrn: failed waiting for nodepool="np-kjsxrn" for cluster "cluster-candidate-4-23-kjsxrn" in resourcegroup="rg-candidate-4-23-9blkbrrd87j2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ea16240d-6235-4ea8-8e63-3873031d6d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ea16240d-6235-4ea8-8e63-3873031d6d51",
      "name": "ea16240d-6235-4ea8-8e63-3873031d6d51",
      "status": "Failed",
      "startTime": "2026-10-08T04:54:34.518182875Z",
      "endTime": "2026-10-08T05:23:35.635646755Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2 to exist\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-6lpm2 to exist\nMachine d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-9blkbrrd87j2-candidate-kjsxrn--managed/providers/Microsoft.Compute/virtualMachines/d9w5y9o4e7p2b1d-np-kjsxrn-79rwc-ct276 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
03:46:27 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5bpm5c" for cluster "cluster-candidate-4-21-5bpm5c"
Unexpected error:
    <*fmt.wrapError | 0x39a1a2b36a20>: 
    failed to create NodePool np-5bpm5c: failed waiting for nodepool="np-5bpm5c" for cluster "cluster-candidate-4-21-5bpm5c" in resourcegroup="rg-candidate-4-21-vptzpz86kwlt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b709679-17d4-47e9-a453-c0f7086a9226
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b709679-17d4-47e9-a453-c0f7086a9226",
      "name": "5b709679-17d4-47e9-a453-c0f7086a9226",
      "status": "Failed",
      "startTime": "2026-10-08T04:50:41.026700325Z",
      "endTime": "2026-10-08T05:19:49.834642675Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99 to exist\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99 to exist\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-5bpm5c" for cluster "cluster-candidate-4-21-5bpm5c"
Unexpected error:
    <*fmt.wrapError | 0x39a1a2b36a20>: 
    failed to create NodePool np-5bpm5c: failed waiting for nodepool="np-5bpm5c" for cluster "cluster-candidate-4-21-5bpm5c" in resourcegroup="rg-candidate-4-21-vptzpz86kwlt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b709679-17d4-47e9-a453-c0f7086a9226
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5b709679-17d4-47e9-a453-c0f7086a9226",
      "name": "5b709679-17d4-47e9-a453-c0f7086a9226",
      "status": "Failed",
      "startTime": "2026-10-08T04:50:41.026700325Z",
      "endTime": "2026-10-08T05:19:49.834642675Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99 to exist\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-8lv99 to exist\nMachine n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-vptzpz86kwlt-candidate-5bpm5c--managed/providers/Microsoft.Compute/virtualMachines/n5d8q2r3o3j4j9i-np-5bpm5c-n48fq-vf7nj to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
2h 23m 35s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
 australiaeast e2e failed n/a 34
e2e (34)
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x3bcb55dd85a0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-crphwg" resourcegroup="rg-np-dg-minor-crphwg-6m9n6b7fcg8d": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-crphwg-6m9n6b7fcg8d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-crphwg/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x3bcb55dd85a0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-crphwg" resourcegroup="rg-np-dg-minor-crphwg-6m9n6b7fcg8d": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-crphwg-6m9n6b7fcg8d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-crphwg/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8tzhmt" for cluster "cluster-candidate-4-22-8tzhmt"
Unexpected error:
    <*fmt.wrapError | 0x23e3b649efa0>: 
    failed to create NodePool np-8tzhmt: failed waiting for nodepool="np-8tzhmt" for cluster "cluster-candidate-4-22-8tzhmt" in resourcegroup="rg-candidate-4-22-sbbrjn549jd7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/33868d99-b424-4bf0-8b0a-debdff616e75
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/33868d99-b424-4bf0-8b0a-debdff616e75",
      "name": "33868d99-b424-4bf0-8b0a-debdff616e75",
      "status": "Failed",
      "startTime": "2026-10-08T02:47:01.121478067Z",
      "endTime": "2026-10-08T03:16:05.005157282Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst to exist\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst to exist\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-8tzhmt" for cluster "cluster-candidate-4-22-8tzhmt"
Unexpected error:
    <*fmt.wrapError | 0x23e3b649efa0>: 
    failed to create NodePool np-8tzhmt: failed waiting for nodepool="np-8tzhmt" for cluster "cluster-candidate-4-22-8tzhmt" in resourcegroup="rg-candidate-4-22-sbbrjn549jd7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/33868d99-b424-4bf0-8b0a-debdff616e75
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/33868d99-b424-4bf0-8b0a-debdff616e75",
      "name": "33868d99-b424-4bf0-8b0a-debdff616e75",
      "status": "Failed",
      "startTime": "2026-10-08T02:47:01.121478067Z",
      "endTime": "2026-10-08T03:16:05.005157282Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst to exist\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-gcgst to exist\nMachine c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-sbbrjn549jd7-candidate-8tzhmt--managed/providers/Microsoft.Compute/virtualMachines/c1c7z9r9e9y0p3n-np-8tzhmt-skdcr-rwlxn to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wlzmmg" for cluster "cluster-candidate-5-1-wlzmmg"
Unexpected error:
    <*fmt.wrapError | 0x1052ae90d2e0>: 
    failed to create NodePool np-wlzmmg: failed waiting for nodepool="np-wlzmmg" for cluster "cluster-candidate-5-1-wlzmmg" in resourcegroup="rg-candidate-5-1-2rb9r54kz678" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/ba8e191c-888b-485b-8866-d49e0334cc7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/ba8e191c-888b-485b-8866-d49e0334cc7c",
      "name": "ba8e191c-888b-485b-8866-d49e0334cc7c",
      "status": "Failed",
      "startTime": "2026-10-08T02:51:25.134077752Z",
      "endTime": "2026-10-08T03:20:27.169387826Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb to exist\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb to exist\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-wlzmmg" for cluster "cluster-candidate-5-1-wlzmmg"
Unexpected error:
    <*fmt.wrapError | 0x1052ae90d2e0>: 
    failed to create NodePool np-wlzmmg: failed waiting for nodepool="np-wlzmmg" for cluster "cluster-candidate-5-1-wlzmmg" in resourcegroup="rg-candidate-5-1-2rb9r54kz678" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/ba8e191c-888b-485b-8866-d49e0334cc7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/ba8e191c-888b-485b-8866-d49e0334cc7c",
      "name": "ba8e191c-888b-485b-8866-d49e0334cc7c",
      "status": "Failed",
      "startTime": "2026-10-08T02:51:25.134077752Z",
      "endTime": "2026-10-08T03:20:27.169387826Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb to exist\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-b2gkb to exist\nMachine y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-1-2rb9r54kz678-candidate-wlzmmg--managed/providers/Microsoft.Compute/virtualMachines/y0d6e1a2j4h5z0e-np-wlzmmg-xvp24-kn8cg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x4fa9c508420>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-s9lxbp6g9wp9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x4fa9c508420>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-s9lxbp6g9wp9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
02:00:37 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2523ecbc0ea0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zgfh95c7gz2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "name": "cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "status": "Failed",
      "startTime": "2026-10-08T03:28:30.373260167Z",
      "endTime": "2026-10-08T03:57:35.369426461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2523ecbc0ea0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zgfh95c7gz2l" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "name": "cc6e942f-2673-4375-ba3b-1fa8f048b004",
      "status": "Failed",
      "startTime": "2026-10-08T03:28:30.373260167Z",
      "endTime": "2026-10-08T03:57:35.369426461Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-dp8nw to exist\nMachine q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zgfh95c7gz2l--managed/providers/Microsoft.Compute/virtualMachines/q0h7k4x2z6s6n0j-np-deletable-tp42z-gfntq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-l5lgrv" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x1c5666c0fce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-hhxjcw4kwjpv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-l5lgrv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-l5lgrv" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x1c5666c0fce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-hhxjcw4kwjpv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-l5lgrv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-bd76nk" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x1ad51ba8c390>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-xg2dpqfdj5bt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-bd76nk
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-bd76nk" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x1ad51ba8c390>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-xg2dpqfdj5bt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-bd76nk
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-kdl52j" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xed02447ca80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-lljlhvtp7gqk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-kdl52j
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-kdl52j" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0xed02447ca80>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-lljlhvtp7gqk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-kdl52j
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tkcz74" for cluster "cluster-candidate-4-21-tkcz74"
Unexpected error:
    <*fmt.wrapError | 0xb57001bd580>: 
    failed to create NodePool np-tkcz74: failed waiting for nodepool="np-tkcz74" for cluster "cluster-candidate-4-21-tkcz74" in resourcegroup="rg-candidate-4-21-qcxhk95mnhvk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/95120e83-d81e-45c6-bda2-21b985646077
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/95120e83-d81e-45c6-bda2-21b985646077",
      "name": "95120e83-d81e-45c6-bda2-21b985646077",
      "status": "Failed",
      "startTime": "2026-10-08T02:45:34.00484297Z",
      "endTime": "2026-10-08T03:14:34.986063906Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx to exist\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx to exist\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-tkcz74" for cluster "cluster-candidate-4-21-tkcz74"
Unexpected error:
    <*fmt.wrapError | 0xb57001bd580>: 
    failed to create NodePool np-tkcz74: failed waiting for nodepool="np-tkcz74" for cluster "cluster-candidate-4-21-tkcz74" in resourcegroup="rg-candidate-4-21-qcxhk95mnhvk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/95120e83-d81e-45c6-bda2-21b985646077
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/95120e83-d81e-45c6-bda2-21b985646077",
      "name": "95120e83-d81e-45c6-bda2-21b985646077",
      "status": "Failed",
      "startTime": "2026-10-08T02:45:34.00484297Z",
      "endTime": "2026-10-08T03:14:34.986063906Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx to exist\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-9fbdx to exist\nMachine u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qcxhk95mnhvk-candidate-tkcz74--managed/providers/Microsoft.Compute/virtualMachines/u8v7l1b3e9b8x1w-np-tkcz74-jqkbr-l669g to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.40 to 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x6b03fb724a0>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-4bzrjj" resourcegroup="rg-np-noedge-4bzrjj-n97rz8gmwdvr": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4bzrjj-n97rz8gmwdvr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-4bzrjj/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.40 to 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x6b03fb724a0>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-4bzrjj" resourcegroup="rg-np-noedge-4bzrjj-n97rz8gmwdvr": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-4bzrjj-n97rz8gmwdvr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-4bzrjj/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0xc813599cf60>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-dz427r" resourcegroup="rg-np-skip-minor-dz427r-crp6tpsxwgq6": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-dz427r-crp6tpsxwgq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-dz427r/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0xc813599cf60>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-dz427r" resourcegroup="rg-np-skip-minor-dz427r-crp6tpsxwgq6": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-dz427r-crp6tpsxwgq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-dz427r/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-64zx6d" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xd58a144ed0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-dstvpqcxjmxk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-64zx6d
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-64zx6d" to 5.0
Unexpected error:
    <*exported.ResponseError | 0xd58a144ed0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-dstvpqcxjmxk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-64zx6d
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
client rate limiter Wait returned an error: context deadline exceeded
02:00:37 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x2c220eff60c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-bxcw427xldqv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/f546f074-ba74-4d4b-ac5f-1b334996fa03
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/f546f074-ba74-4d4b-ac5f-1b334996fa03",
      "name": "f546f074-ba74-4d4b-ac5f-1b334996fa03",
      "status": "Failed",
      "startTime": "2026-10-08T02:18:39.603761603Z",
      "endTime": "2026-10-08T02:47:44.845130459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-cxd7w-m4vt2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bxcw427xldqv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-cxd7w-m4vt2 to exist\nMachine agg-cluster-agg-np-a-cxd7w-th9qk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-cxd7w-th9qk: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-cxd7w-m4vt2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bxcw427xldqv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-cxd7w-m4vt2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0x2c220eff60c0>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-bxcw427xldqv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/f546f074-ba74-4d4b-ac5f-1b334996fa03
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/f546f074-ba74-4d4b-ac5f-1b334996fa03",
      "name": "f546f074-ba74-4d4b-ac5f-1b334996fa03",
      "status": "Failed",
      "startTime": "2026-10-08T02:18:39.603761603Z",
      "endTime": "2026-10-08T02:47:44.845130459Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want \u003e= 2 (autoscaling min); node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine agg-cluster-agg-np-a-cxd7w-m4vt2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bxcw427xldqv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-cxd7w-m4vt2 to exist\nMachine agg-cluster-agg-np-a-cxd7w-th9qk: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine agg-cluster-agg-np-a-cxd7w-th9qk: Creating: virtualmachine creating or updating\nMachine agg-cluster-agg-np-a-cxd7w-m4vt2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-bxcw427xldqv--managed/providers/Microsoft.Compute/virtualMachines/agg-cluster-agg-np-a-cxd7w-m4vt2 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0x26e1bcba84a0>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-7jfzmq" resourcegroup="rg-np-downgrade-7jfzmq-rvr22sztdn98": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-7jfzmq-rvr22sztdn98/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-7jfzmq/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0x26e1bcba84a0>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-7jfzmq" resourcegroup="rg-np-downgrade-7jfzmq-rvr22sztdn98": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-7jfzmq-rvr22sztdn98/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-7jfzmq/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-hn2cnv" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x9116922ea20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-4w8m65kzp5nn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-hn2cnv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-hn2cnv" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0x9116922ea20>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-4w8m65kzp5nn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-hn2cnv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-xjwrx8" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x3f5c96b45080>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-lt88zk4qvv8g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-xjwrx8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-xjwrx8" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x3f5c96b45080>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-lt88zk4qvv8g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-xjwrx8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x28f92e3a99c0>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-tw9ncc" resourcegroup="rg-np-version-upgrade-tw9ncc-98zjd59jkjpr": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tw9ncc-98zjd59jkjpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tw9ncc/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x28f92e3a99c0>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-tw9ncc" resourcegroup="rg-np-version-upgrade-tw9ncc-98zjd59jkjpr": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tw9ncc-98zjd59jkjpr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tw9ncc/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:37 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x65e48efd800>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-qwp7bbpclw7g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1e1feb34-d8de-476b-993b-80f75f40bd9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1e1feb34-d8de-476b-993b-80f75f40bd9e",
      "name": "1e1feb34-d8de-476b-993b-80f75f40bd9e",
      "status": "Failed",
      "startTime": "2026-10-08T02:19:11.90853804Z",
      "endTime": "2026-10-08T02:48:14.711261391Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-p24md: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-p24md to exist\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-p24md: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-p24md to exist\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x65e48efd800>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-qwp7bbpclw7g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1e1feb34-d8de-476b-993b-80f75f40bd9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/1e1feb34-d8de-476b-993b-80f75f40bd9e",
      "name": "1e1feb34-d8de-476b-993b-80f75f40bd9e",
      "status": "Failed",
      "startTime": "2026-10-08T02:19:11.90853804Z",
      "endTime": "2026-10-08T02:48:14.711261391Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-p24md: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-p24md to exist\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-p24md: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-p24md to exist\nMachine e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-qwp7bbpclw7g--managed/providers/Microsoft.Compute/virtualMachines/e1v5o5b0z6h4y7i-np-1-xh5kh-q4hwl to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-x2nmgc" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x3577745b47e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-l49w7x6sf8lk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-x2nmgc
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-x2nmgc" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x3577745b47e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-l49w7x6sf8lk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-x2nmgc
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x35b2e2ff91e0>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-rx2rkxms8sw7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-rx2rkxms8sw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0x35b2e2ff91e0>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-rx2rkxms8sw7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-rx2rkxms8sw7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-wps54w" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x3ddfbef4d200>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-9qcq2bc8pfhn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-wps54w
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-wps54w" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x3ddfbef4d200>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-9qcq2bc8pfhn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-wps54w
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-znb5qj" for cluster "cluster-candidate-5-0-znb5qj"
Unexpected error:
    <*fmt.wrapError | 0x17bbc4694220>: 
    failed to create NodePool np-znb5qj: failed waiting for nodepool="np-znb5qj" for cluster "cluster-candidate-5-0-znb5qj" in resourcegroup="rg-candidate-5-0-d5trjc74qhsj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/14de4838-40ca-415c-a29b-7d78f06ecdac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/14de4838-40ca-415c-a29b-7d78f06ecdac",
      "name": "14de4838-40ca-415c-a29b-7d78f06ecdac",
      "status": "Failed",
      "startTime": "2026-10-08T02:49:59.670336775Z",
      "endTime": "2026-10-08T03:19:07.230773535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82 to exist\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82 to exist\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-znb5qj" for cluster "cluster-candidate-5-0-znb5qj"
Unexpected error:
    <*fmt.wrapError | 0x17bbc4694220>: 
    failed to create NodePool np-znb5qj: failed waiting for nodepool="np-znb5qj" for cluster "cluster-candidate-5-0-znb5qj" in resourcegroup="rg-candidate-5-0-d5trjc74qhsj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/14de4838-40ca-415c-a29b-7d78f06ecdac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/14de4838-40ca-415c-a29b-7d78f06ecdac",
      "name": "14de4838-40ca-415c-a29b-7d78f06ecdac",
      "status": "Failed",
      "startTime": "2026-10-08T02:49:59.670336775Z",
      "endTime": "2026-10-08T03:19:07.230773535Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82 to exist\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-gpw82 to exist\nMachine l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-d5trjc74qhsj-candidate-znb5qj--managed/providers/Microsoft.Compute/virtualMachines/l1n2h0z1b3o2j0p-np-znb5qj-cx29j-sxz4v to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeploymentFailed; detail code NotFound; detail message Resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.; provider Microsoft.ManagedIdentity
02:00:37 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:101]: failed to deploy dedicated managed identities
Unexpected error:
    <*fmt.wrapError | 0x2a6c3876d160>: 
    failed waiting for deployment "mi-missing-mi-hcp-cluster-8qb82d" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-missing-mi-hcp-cluster-8qb82d/operationStatuses/08584101764787793835
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterNonMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:101]: failed to deploy dedicated managed identities
Unexpected error:
    <*fmt.wrapError | 0x2a6c3876d160>: 
    failed waiting for deployment "mi-missing-mi-hcp-cluster-8qb82d" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-missing-mi-hcp-cluster-8qb82d/operationStatuses/08584101764787793835
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterNonMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-controller-manager-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cloud-network-config-missing-mi-hcp-cluster' was not found.\"\r\n          },\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.Resources/deployments/clusterMsiScopedAssignments-jaooh6lj24n4o\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"NotFound\",\r\n            \"message\": \"Resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/delete-mi-rg-zm56p642vwdr/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service-missing-mi-hcp-cluster' was not found.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x6f28c1d0c00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vpk5q2z5574v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x6f28c1d0c00>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vpk5q2z5574v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x137051db4300>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-hcx5lwfp6892/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x137051db4300>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-hcx5lwfp6892/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xhn5bs" for cluster "cluster-candidate-4-23-xhn5bs"
Unexpected error:
    <*fmt.wrapError | 0x31d2ef35d8e0>: 
    failed to create NodePool np-xhn5bs: failed waiting for nodepool="np-xhn5bs" for cluster "cluster-candidate-4-23-xhn5bs" in resourcegroup="rg-candidate-4-23-4llsqxjhkgt4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/93da1929-fde9-4a74-85a6-ad361a679f40
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/93da1929-fde9-4a74-85a6-ad361a679f40",
      "name": "93da1929-fde9-4a74-85a6-ad361a679f40",
      "status": "Failed",
      "startTime": "2026-10-08T02:49:02.43694261Z",
      "endTime": "2026-10-08T03:18:07.302711356Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr to exist\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr to exist\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-xhn5bs" for cluster "cluster-candidate-4-23-xhn5bs"
Unexpected error:
    <*fmt.wrapError | 0x31d2ef35d8e0>: 
    failed to create NodePool np-xhn5bs: failed waiting for nodepool="np-xhn5bs" for cluster "cluster-candidate-4-23-xhn5bs" in resourcegroup="rg-candidate-4-23-4llsqxjhkgt4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/93da1929-fde9-4a74-85a6-ad361a679f40
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/93da1929-fde9-4a74-85a6-ad361a679f40",
      "name": "93da1929-fde9-4a74-85a6-ad361a679f40",
      "status": "Failed",
      "startTime": "2026-10-08T02:49:02.43694261Z",
      "endTime": "2026-10-08T03:18:07.302711356Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr to exist\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-bjrwr to exist\nMachine t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-4llsqxjhkgt4-candidate-xhn5bs--managed/providers/Microsoft.Compute/virtualMachines/t7o7j7t1z4g4t0m-np-xhn5bs-lm5cm-gcr5k to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2edd5c272b00>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-df62qq" resourcegroup="rg-np-version-upgrade-df62qq-l9cc8d44b2p9": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-df62qq-l9cc8d44b2p9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-df62qq/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2edd5c272b00>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-df62qq" resourcegroup="rg-np-version-upgrade-df62qq-l9cc8d44b2p9": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-df62qq-l9cc8d44b2p9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-df62qq/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:37 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x1b63d32ea960>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-6l2qnmrqmq6s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/987e3697-e775-428f-b96b-007cf98cd200
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/987e3697-e775-428f-b96b-007cf98cd200",
      "name": "987e3697-e775-428f-b96b-007cf98cd200",
      "status": "Failed",
      "startTime": "2026-10-08T02:17:06.289611141Z",
      "endTime": "2026-10-08T02:46:14.822515799Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-b4b48-6xc2t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-6l2qnmrqmq6s--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-b4b48-6xc2t to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-b4b48-6xc2t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-6l2qnmrqmq6s--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-b4b48-6xc2t to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:159]: failed to create node pool "np-1" for private KAS cluster "private-kas"
Unexpected error:
    <*fmt.wrapError | 0x1b63d32ea960>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kas" in resourcegroup="private-kas-6l2qnmrqmq6s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/987e3697-e775-428f-b96b-007cf98cd200
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/987e3697-e775-428f-b96b-007cf98cd200",
      "name": "987e3697-e775-428f-b96b-007cf98cd200",
      "status": "Failed",
      "startTime": "2026-10-08T02:17:06.289611141Z",
      "endTime": "2026-10-08T02:46:14.822515799Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 1; node pool condition AllNodesHealthy is False: 1 of 1 machines are not healthy\nMachine private-kas-np-1-b4b48-6xc2t: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-6l2qnmrqmq6s--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-b4b48-6xc2t to exist\n; node pool condition AllMachinesReady is False: 1 of 1 machines are not ready\nMachine private-kas-np-1-b4b48-6xc2t: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-6l2qnmrqmq6s--managed/providers/Microsoft.Compute/virtualMachines/private-kas-np-1-b4b48-6xc2t to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-xf8crr" to 4.23
Unexpected error:
    <*exported.ResponseError | 0x28f031e29260>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-mqfd4qzn4578/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xf8crr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-xf8crr" to 4.23
Unexpected error:
    <*exported.ResponseError | 0x28f031e29260>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-mqfd4qzn4578/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-xf8crr
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:37 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xe292679e030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-fwdf5gjhw97g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e74aa6d0-c8cd-430c-8190-549622080499
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e74aa6d0-c8cd-430c-8190-549622080499",
      "name": "e74aa6d0-c8cd-430c-8190-549622080499",
      "status": "Failed",
      "startTime": "2026-10-08T02:18:33.237818266Z",
      "endTime": "2026-10-08T02:47:34.745399444Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-dc67w-8c8mq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-8c8mq to exist\nMachine cilium-cluster-cilium-np-dc67w-p8qqc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-p8qqc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-dc67w-8c8mq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-8c8mq to exist\nMachine cilium-cluster-cilium-np-dc67w-p8qqc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-p8qqc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xe292679e030>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: no cilium pods found in kube-system namespace
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-fwdf5gjhw97g" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e74aa6d0-c8cd-430c-8190-549622080499
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/e74aa6d0-c8cd-430c-8190-549622080499",
      "name": "e74aa6d0-c8cd-430c-8190-549622080499",
      "status": "Failed",
      "startTime": "2026-10-08T02:18:33.237818266Z",
      "endTime": "2026-10-08T02:47:34.745399444Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-dc67w-8c8mq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-8c8mq to exist\nMachine cilium-cluster-cilium-np-dc67w-p8qqc: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-p8qqc to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-dc67w-8c8mq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-8c8mq to exist\nMachine cilium-cluster-cilium-np-dc67w-p8qqc: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-fwdf5gjhw97g--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-dc67w-p8qqc to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.23 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-pcrbcd" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x33c001cb49c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-zb4tmg42vgq7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-pcrbcd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-pcrbcd" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x33c001cb49c0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-zb4tmg42vgq7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-pcrbcd
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x19c5b50bc860>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-x2784j5xhvrh": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-x2784j5xhvrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x19c5b50bc860>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-x2784j5xhvrh": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-x2784j5xhvrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
02:00:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-gm6fx9" for cluster "cluster-candidate-4-20-gm6fx9"
Unexpected error:
    <*fmt.wrapError | 0x130c0be607c0>: 
    failed to create NodePool np-gm6fx9: failed waiting for nodepool="np-gm6fx9" for cluster "cluster-candidate-4-20-gm6fx9" in resourcegroup="rg-candidate-4-20-ts84fqwf8d5v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9435cae-155f-47ef-8d90-e886b71aff15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9435cae-155f-47ef-8d90-e886b71aff15",
      "name": "b9435cae-155f-47ef-8d90-e886b71aff15",
      "status": "Failed",
      "startTime": "2026-10-08T02:45:36.63094836Z",
      "endTime": "2026-10-08T03:14:44.806443674Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj to exist\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj to exist\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-gm6fx9" for cluster "cluster-candidate-4-20-gm6fx9"
Unexpected error:
    <*fmt.wrapError | 0x130c0be607c0>: 
    failed to create NodePool np-gm6fx9: failed waiting for nodepool="np-gm6fx9" for cluster "cluster-candidate-4-20-gm6fx9" in resourcegroup="rg-candidate-4-20-ts84fqwf8d5v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9435cae-155f-47ef-8d90-e886b71aff15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/australiaeast/hcpOperationStatuses/b9435cae-155f-47ef-8d90-e886b71aff15",
      "name": "b9435cae-155f-47ef-8d90-e886b71aff15",
      "status": "Failed",
      "startTime": "2026-10-08T02:45:36.63094836Z",
      "endTime": "2026-10-08T03:14:44.806443674Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj to exist\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7c4lj to exist\nMachine p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-ts84fqwf8d5v-candidate-gm6fx9--managed/providers/Microsoft.Compute/virtualMachines/p6i3y8w1a7s1v4y-np-gm6fx9-xdhhm-7vpdv to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
02:00:37 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xa63e00795e0>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-dwrcvf" resourcegroup="rg-np-version-upgrade-dwrcvf-tndq67k8xt89": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-dwrcvf-tndq67k8xt89/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-dwrcvf/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0xa63e00795e0>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-dwrcvf" resourcegroup="rg-np-version-upgrade-dwrcvf-tndq67k8xt89": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-dwrcvf-tndq67k8xt89/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-dwrcvf/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred